{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "customizing_adanet.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "q4WF3l23pumU"
      },
      "source": [
        "##### Copyright 2018 The AdaNet Authors."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "both",
        "colab_type": "code",
        "id": "Kic2quJWppmx",
        "colab": {
          "test": {
            "output": "ignore",
            "timeout": 900
          }
        }
      },
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "aL7SpaKdirqG"
      },
      "source": [
        "# Customizing AdaNet"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "i5s1gsS1bOuB"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/adanet/blob/master/adanet/examples/tutorials/customizing_adanet.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/adanet/blob/master/adanet/examples/tutorials/customizing_adanet.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "</table>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ySKpPtPrmaCx"
      },
      "source": [
        "Often times, as a researcher or machine learning practitioner, you will have\n",
        "some prior knowledge about a dataset. Ideally you should be able to encode that\n",
        "knowledge into your machine learning algorithm. With `adanet`, you can do so by\n",
        "defining the *neural architecture search space* that the AdaNet algorithm should\n",
        "explore.\n",
        "\n",
        "In this tutorial, we will explore the flexibility of the `adanet` framework, and\n",
        "create a custom search space for an image-classification dataset using high-level\n",
        "TensorFlow libraries like the\n",
        "[`tf.keras.layers`](https://www.tensorflow.org/guide/keras#build_advanced_models)\n",
        "functional API.\n",
        "\n"]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "xB1akik24RFa",
        "outputId": "426b7e06-c0e0-42f5-f271-a53e92799305",
        "colab": {
          "test": {
            "output": "ignore",
            "timeout": 900
          },
          "base_uri": "https://localhost:8080/",
          "height": 683
        }
      },
      "source": [
        "#@test {\"skip\": true}\n",
        "# If you're running this in Colab, first install the adanet package:\n",
        "!pip install adanet"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting adanet\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/43/a1/4045b9cdb2e90987dac61ed8da2b66ca3e42c866232698b51de178f2184d/adanet-0.8.0-py2.py3-none-any.whl (117kB)\n",
            "\u001b[K     |████████████████████████████████| 122kB 2.7MB/s \n",
            "\u001b[?25hRequirement already satisfied: protobuf<4.0,>=3.6 in /usr/local/lib/python3.6/dist-packages (from adanet) (3.10.0)\n",
            "Collecting rednose<2.0,>=1.3\n",
            "  Downloading https://files.pythonhosted.org/packages/3a/a8/4b73ae7466c2e9b63b3c4d66040d1c0eda1f764812353753702546d8c87f/rednose-1.3.0.tar.gz\n",
            "Requirement already satisfied: absl-py<1.0,>=0.7 in /usr/local/lib/python3.6/dist-packages (from adanet) (0.9.0)\n",
            "Collecting mock<4.0,>=3.0\n",
            "  Downloading https://files.pythonhosted.org/packages/05/d2/f94e68be6b17f46d2c353564da56e6fb89ef09faeeff3313a046cb810ca9/mock-3.0.5-py2.py3-none-any.whl\n",
            "Collecting nose<2.0,>=1.3\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/15/d8/dd071918c040f50fa1cf80da16423af51ff8ce4a0f2399b7bf8de45ac3d9/nose-1.3.7-py3-none-any.whl (154kB)\n",
            "\u001b[K     |████████████████████████████████| 163kB 8.6MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy<2.0,>=1.15 in /usr/local/lib/python3.6/dist-packages (from adanet) (1.17.5)\n",
            "Collecting coverage<5.0,>=4.5\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/51/b1/13609068fff1c8c056f0c4601ad6985cf5c1bbfc529196ab08bd2a57dc39/coverage-4.5.4-cp36-cp36m-manylinux1_x86_64.whl (205kB)\n",
            "\u001b[K     |████████████████████████████████| 215kB 49.4MB/s \n",
            "\u001b[?25hRequirement already satisfied: six<2.0,>=1.11 in /usr/local/lib/python3.6/dist-packages (from adanet) (1.12.0)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf<4.0,>=3.6->adanet) (42.0.2)\n",
            "Collecting termstyle>=0.1.7\n",
            "  Downloading https://files.pythonhosted.org/packages/65/53/4dfdfe12b499f375cc78caca9cf146c01e752bab7713de4510d35e3da306/termstyle-0.1.11.tar.gz\n",
            "Collecting colorama\n",
            "  Downloading https://files.pythonhosted.org/packages/c9/dc/45cdef1b4d119eb96316b3117e6d5708a08029992b2fee2c143c7a0a5cc5/colorama-0.4.3-py2.py3-none-any.whl\n",
            "Building wheels for collected packages: rednose, termstyle\n",
            "  Building wheel for rednose (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for rednose: filename=rednose-1.3.0-cp36-none-any.whl size=12155 sha256=4a499251deaec6c7adfa222e583b7baf3f508a2b3c95db792f3def0fd2e2db34\n",
            "  Stored in directory: /root/.cache/pip/wheels/21/be/74/e74fbccdaf7ab983f847153b2708e4d1722801ce7265dfa643\n",
            "  Building wheel for termstyle (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for termstyle: filename=termstyle-0.1.11-cp36-none-any.whl size=4778 sha256=1d175ec8628584ad81710cb35040e6cbd6b25d3d5aae0fba644840a8f58fc446\n",
            "  Stored in directory: /root/.cache/pip/wheels/93/0c/cf/1d134a42237c338ee1d733c81b92d95f6d04d9531bf4c2a9a1\n",
            "Successfully built rednose termstyle\n",
            "\u001b[31mERROR: datascience 0.10.6 has requirement coverage==3.7.1, but you'll have coverage 4.5.4 which is incompatible.\u001b[0m\n",
            "\u001b[31mERROR: datascience 0.10.6 has requirement folium==0.2.1, but you'll have folium 0.8.3 which is incompatible.\u001b[0m\n",
            "\u001b[31mERROR: coveralls 0.5 has requirement coverage<3.999,>=3.6, but you'll have coverage 4.5.4 which is incompatible.\u001b[0m\n",
            "Installing collected packages: termstyle, colorama, rednose, mock, nose, coverage, adanet\n",
            "  Found existing installation: coverage 3.7.1\n",
            "    Uninstalling coverage-3.7.1:\n",
            "      Successfully uninstalled coverage-3.7.1\n",
            "Successfully installed adanet-0.8.0 colorama-0.4.3 coverage-4.5.4 mock-3.0.5 nose-1.3.7 rednose-1.3.0 termstyle-0.1.11\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "x_3b6xx2s6B9",
        "outputId": "3faf4255-484d-4fa3-942e-7b3fe0a6796d",
        "colab": {
          "test": {
            "output": "ignore",
            "timeout": 900
          },
          "base_uri": "https://localhost:8080/",
          "height": 151
        }
      },
      "source": [
        "import functools\n",
        "import os\n",
        "import shutil\n",
        "\n",
        "import adanet\n",
        "from adanet.examples import simple_dnn\n",
        "import matplotlib.pyplot as plt\n",
        "import tensorflow.compat.v1 as tf\n",
        "\n",
        "\n",
        "# The random seed to use.\n",
        "RANDOM_SEED = 42\n",
        "\n",
        "LOG_DIR = '/tmp/models'"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<p style=\"color: red;\">\n",
              "The default version of TensorFlow in Colab will soon switch to TensorFlow 2.x.<br>\n",
              "We recommend you <a href=\"https://www.tensorflow.org/guide/migrate\" target=\"_blank\">upgrade</a> now \n",
              "or ensure your notebook will continue to use TensorFlow 1.x via the <code>%tensorflow_version 1.x</code> magic:\n",
              "<a href=\"https://colab.research.google.com/notebooks/tensorflow_version.ipynb\" target=\"_blank\">more info</a>.</p>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/adanet/tf_compat/__init__.py:87: The name tf.losses.Reduction is deprecated. Please use tf.compat.v1.losses.Reduction instead.\n",
            "\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/adanet/core/tpu_estimator.py:39: The name tf.estimator.tpu.TPUEstimator is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimator instead.\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "7gE5Mm9j2oYw"
      },
      "source": [
        "## Fashion MNIST dataset\n",
        "\n",
        "In this example, we will use the Fashion MNIST dataset\n",
        "[[Xiao et al., 2017](https://arxiv.org/abs/1708.07747)] for classifying fashion\n",
        "apparel images into one of ten categories:\n",
        "\n",
        "1.  T-shirt/top\n",
        "2.  Trouser\n",
        "3.  Pullover\n",
        "4.  Dress\n",
        "5.  Coat\n",
        "6.  Sandal\n",
        "7.  Shirt\n",
        "8.  Sneaker\n",
        "9.  Bag\n",
        "10. Ankle boot\n",
        "\n",
        "![Fashion MNIST](https://github.com/zalandoresearch/fashion-mnist/blob/master/doc/img/fashion-mnist-sprite.png?raw=true)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "5_hRtdchqRZb"
      },
      "source": [
        "## Download the data\n",
        "\n",
        "Conveniently, the data is available via Keras:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "uYklOnPJ4h7g",
        "outputId": "bc3b7356-3a29-4d76-842b-6d5c8b163d85",
        "colab": {
          "test": {
            "output": "ignore",
            "timeout": 900
          },
          "base_uri": "https://localhost:8080/",
          "height": 153
        }
      },
      "source": [
        "(x_train, y_train), (x_test, y_test) = (\n",
        "    tf.keras.datasets.fashion_mnist.load_data())"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-labels-idx1-ubyte.gz\n",
            "32768/29515 [=================================] - 0s 0us/step\n",
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-images-idx3-ubyte.gz\n",
            "26427392/26421880 [==============================] - 0s 0us/step\n",
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-labels-idx1-ubyte.gz\n",
            "8192/5148 [===============================================] - 0s 0us/step\n",
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-images-idx3-ubyte.gz\n",
            "4423680/4422102 [==============================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "tECo5dFd4QCa"
      },
      "source": [
        "## Supply the data in TensorFlow\n",
        "\n",
        "Our first task is to supply the data in TensorFlow. Using the\n",
        "tf.estimator.Estimator covention, we will define a function that returns an\n",
        "`input_fn` which returns feature and label `Tensors`.\n",
        "\n",
        "We will also use the `tf.data.Dataset` API to feed the data into our models."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "gxTAoIXwsTH7",
        "colab": {
          "test": {
            "output": "ignore",
            "timeout": 900
          }
        }
      },
      "source": [
        "FEATURES_KEY = \"images\"\n",
        "\n",
        "\n",
        "def generator(images, labels):\n",
        "  \"\"\"Returns a generator that returns image-label pairs.\"\"\"\n",
        "\n",
        "  def _gen():\n",
        "    for image, label in zip(images, labels):\n",
        "      yield image, label\n",
        "\n",
        "  return _gen\n",
        "\n",
        "\n",
        "def preprocess_image(image, label):\n",
        "  \"\"\"Preprocesses an image for an `Estimator`.\"\"\"\n",
        "  # First let's scale the pixel values to be between 0 and 1.\n",
        "  image = image / 255.\n",
        "  # Next we reshape the image so that we can apply a 2D convolution to it.\n",
        "  image = tf.reshape(image, [28, 28, 1])\n",
        "  # Finally the features need to be supplied as a dictionary.\n",
        "  features = {FEATURES_KEY: image}\n",
        "  return features, label\n",
        "\n",
        "\n",
        "def input_fn(partition, training, batch_size):\n",
        "  \"\"\"Generate an input_fn for the Estimator.\"\"\"\n",
        "\n",
        "  def _input_fn():\n",
        "    if partition == \"train\":\n",
        "      dataset = tf.data.Dataset.from_generator(\n",
        "          generator(x_train, y_train), (tf.float32, tf.int32), ((28, 28), ()))\n",
        "    elif partition == \"predict\":\n",
        "      dataset = tf.data.Dataset.from_generator(\n",
        "          generator(x_test[:10], y_test[:10]), (tf.float32, tf.int32), ((28,28), ()))\n",
        "    else:\n",
        "      dataset = tf.data.Dataset.from_generator(\n",
        "          generator(x_test, y_test), (tf.float32, tf.int32), ((28, 28), ()))\n",
        "\n",
        "    # We call repeat after shuffling, rather than before, to prevent separate\n",
        "    # epochs from blending together.\n",
        "    if training:\n",
        "      dataset = dataset.shuffle(10 * batch_size, seed=RANDOM_SEED).repeat()\n",
        "\n",
        "    dataset = dataset.map(preprocess_image).batch(batch_size)\n",
        "    iterator = dataset.make_one_shot_iterator()\n",
        "    features, labels = iterator.get_next()\n",
        "    return features, labels\n",
        "\n",
        "  return _input_fn"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "fhu2zpf9faIB"
      },
      "source": [
        "## Launch TensorBoard\n",
        "\n",
        "Let's run [TensorBoard](https://www.tensorflow.org/guide/summaries_and_tensorboard) to visualize model training over time. We'll use [ngrok](https://ngrok.com/) to tunnel traffic to localhost.\n",
        "\n",
        "*The instructions for setting up Tensorboard were obtained from https://www.dlology.com/blog/quick-guide-to-run-tensorboard-in-google-colab/*\n",
        "\n",
        "Run the next cells and follow the link to see the TensorBoard in a new tab."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "S1UcP5yeaDz9",
        "outputId": "aace39a4-81b4-47f1-f057-8693c67c9efd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 306
        }
      },
      "source": [
        "#@test {\"skip\": true}\n",
        "\n",
        "get_ipython().system_raw(\n",
        "    'tensorboard --logdir {} --host 0.0.0.0 --port 6006 &'\n",
        "    .format(LOG_DIR)\n",
        ")\n",
        "\n",
        "# Install ngrok binary.\n",
        "! wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip\n",
        "! unzip ngrok-stable-linux-amd64.zip\n",
        "\n",
        "# Delete old logs dir.\n",
        "shutil.rmtree(LOG_DIR, ignore_errors=True)\n",
        "\n",
        "print(\"Follow this link to open TensorBoard in a new tab.\")\n",
        "get_ipython().system_raw('./ngrok http 6006 &')\n",
        "! curl -s http://localhost:4040/api/tunnels | python3 -c \\\n",
        "    \"import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])\"\n",
        "\n"],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-01-28 23:24:13--  https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip\n",
            "Resolving bin.equinox.io (bin.equinox.io)... 34.238.36.128, 34.193.139.214, 34.206.126.139, ...\n",
            "Connecting to bin.equinox.io (bin.equinox.io)|34.238.36.128|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 13773305 (13M) [application/octet-stream]\n",
            "Saving to: ‘ngrok-stable-linux-amd64.zip’\n",
            "\n",
            "ngrok-stable-linux- 100%[===================>]  13.13M  6.62MB/s    in 2.0s    \n",
            "\n",
            "2020-01-28 23:24:15 (6.62 MB/s) - ‘ngrok-stable-linux-amd64.zip’ saved [13773305/13773305]\n",
            "\n",
            "Archive:  ngrok-stable-linux-amd64.zip\n",
            "  inflating: ngrok                   \n",
            "Follow this link to open TensorBoard in a new tab.\n",
            "Traceback (most recent call last):\n",
            "  File \"<string>\", line 1, in <module>\n",
            "IndexError: list index out of range\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "vm9yudEv5lQZ"
      },
      "source": [
        "## Establish baselines\n",
        "\n",
        "The next task should be to get somes baselines to see how our model performs on\n",
        "this dataset.\n",
        "\n",
        "Let's define some information to share with all our `tf.estimator.Estimators`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "xNwSUWh-9_Ib",
        "colab": {
          "test": {
            "output": "ignore",
            "timeout": 900
          }
        }
      },
      "source": [
        "# The number of classes.\n",
        "NUM_CLASSES = 10\n",
        "\n",
        "# A `Head` instance defines the loss function and metrics for `Estimators`.\n",
        "head = tf.estimator.MultiClassHead(NUM_CLASSES)\n",
        "\n",
        "# Some `Estimators` use feature columns for understanding their input features.\n",
        "feature_columns = [\n",
        "    tf.feature_column.numeric_column(FEATURES_KEY, shape=[28, 28, 1])\n",
        "]\n",
        "\n",
        "def make_config(experiment_name):\n",
        "  # Estimator configuration.\n",
        "  return tf.estimator.RunConfig(\n",
        "    save_checkpoints_steps=1000,\n",
        "    save_summary_steps=1000,\n",
        "    tf_random_seed=RANDOM_SEED,\n",
        "    model_dir=os.path.join(LOG_DIR, experiment_name))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "QY0cv-ot-Gxs"
      },
      "source": [
        "Let's start simple, and train a linear model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "both",
        "colab_type": "code",
        "id": "s8wJKsi06blX",
        "outputId": "786f17b6-4a14-4244-fe64-ba2365392130",
        "colab": {
          "test": {
            "output": "ignore",
            "timeout": 900
          },
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "#@test {\"skip\": true}\n",
        "#@title Parameters\n",
        "LEARNING_RATE = 0.001  #@param {type:\"number\"}\n",
        "TRAIN_STEPS = 5000  #@param {type:\"integer\"}\n",
        "BATCH_SIZE = 64  #@param {type:\"integer\"}\n",
        "\n",
        "estimator = tf.estimator.LinearClassifier(\n",
        "    feature_columns=feature_columns,\n",
        "    n_classes=NUM_CLASSES,\n",
        "    optimizer=tf.train.RMSPropOptimizer(learning_rate=LEARNING_RATE),\n",
        "    config=make_config(\"linear\"))\n",
        "\n",
        "tf.estimator.train_and_evaluate(\n",
        "    estimator,\n",
        "    train_spec=tf.estimator.TrainSpec(\n",
        "        input_fn=input_fn(\"train\", training=True, batch_size=BATCH_SIZE),\n",
        "        max_steps=TRAIN_STEPS),\n",
        "    eval_spec=tf.estimator.EvalSpec(\n",
        "        input_fn=input_fn(\"test\", training=False, batch_size=BATCH_SIZE),\n",
        "        steps=None,\n",
        "        start_delay_secs=1,\n",
        "        throttle_secs=1,  \n",
        "    ))\n",
        "results = estimator.evaluate(\n",
        "    input_fn(\"test\", training=False, batch_size=BATCH_SIZE),\n",
        "    steps=None)\n",
        "print(\"Accuracy:\", results[\"accuracy\"])\n",
        "print(\"Loss:\", results[\"average_loss\"])"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/linear', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b1c3ec080>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
            "INFO:tensorflow:Not using Distribute Coordinator.\n",
            "INFO:tensorflow:Running training and evaluation locally (non-distributed).\n",
            "INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 1000 or save_checkpoints_secs None.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/util/random_seed.py:58: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
            "WARNING:tensorflow:From <ipython-input-5-2cc59c672b77>:45: DatasetV1.make_one_shot_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `for ... in dataset:` to iterate over a dataset. If using `tf.estimator`, return the `Dataset` object directly from your input function. As a last resort, you can use `tf.compat.v1.data.make_one_shot_iterator(dataset)`.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/feature_column/feature_column_v2.py:305: Layer.add_variable (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use `layer.add_weight` method instead.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "If using Keras pass *_constraint arguments to layers.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/canned/linear.py:308: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.cast` instead.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/rmsprop.py:119: calling Ones.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Call initializer instance with the dtype argument instead of passing it to the constructor\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Create CheckpointSaverHook.\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving checkpoints for 0 into /tmp/models/linear/model.ckpt.\n",
            "INFO:tensorflow:loss = 147.36546, step = 1\n",
            "INFO:tensorflow:global_step/sec: 72.6272\n",
            "INFO:tensorflow:loss = 45.435265, step = 101 (1.385 sec)\n",
            "INFO:tensorflow:global_step/sec: 72.226\n",
            "INFO:tensorflow:loss = 47.006966, step = 201 (1.380 sec)\n",
            "INFO:tensorflow:global_step/sec: 69.7211\n",
            "INFO:tensorflow:loss = 29.17189, step = 301 (1.438 sec)\n",
            "INFO:tensorflow:global_step/sec: 69.9187\n",
            "INFO:tensorflow:loss = 39.85762, step = 401 (1.428 sec)\n",
            "INFO:tensorflow:global_step/sec: 69.1659\n",
            "INFO:tensorflow:loss = 26.399818, step = 501 (1.442 sec)\n",
            "INFO:tensorflow:global_step/sec: 69.4258\n",
            "INFO:tensorflow:loss = 45.849102, step = 601 (1.441 sec)\n",
            "INFO:tensorflow:global_step/sec: 73.9524\n",
            "INFO:tensorflow:loss = 47.62218, step = 701 (1.352 sec)\n",
            "INFO:tensorflow:global_step/sec: 74.9002\n",
            "INFO:tensorflow:loss = 36.039284, step = 801 (1.335 sec)\n",
            "INFO:tensorflow:global_step/sec: 73.5437\n",
            "INFO:tensorflow:loss = 20.321949, step = 901 (1.359 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 1000 into /tmp/models/linear/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2020-01-28T23:24:37Z\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/linear/model.ckpt-1000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Finished evaluation at 2020-01-28-23:24:39\n",
            "INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.8208, average_loss = 0.5218894, global_step = 1000, loss = 33.241364\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmp/models/linear/model.ckpt-1000\n",
            "INFO:tensorflow:global_step/sec: 24.0609\n",
            "INFO:tensorflow:loss = 33.510307, step = 1001 (4.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 69.5201\n",
            "INFO:tensorflow:loss = 30.549637, step = 1101 (1.432 sec)\n",
            "INFO:tensorflow:global_step/sec: 70.3965\n",
            "INFO:tensorflow:loss = 39.326317, step = 1201 (1.423 sec)\n",
            "INFO:tensorflow:global_step/sec: 71.8143\n",
            "INFO:tensorflow:loss = 29.402462, step = 1301 (1.395 sec)\n",
            "INFO:tensorflow:global_step/sec: 71.0009\n",
            "INFO:tensorflow:loss = 37.390305, step = 1401 (1.404 sec)\n",
            "INFO:tensorflow:global_step/sec: 71.2378\n",
            "INFO:tensorflow:loss = 33.978073, step = 1501 (1.404 sec)\n",
            "INFO:tensorflow:global_step/sec: 70.2144\n",
            "INFO:tensorflow:loss = 23.10869, step = 1601 (1.425 sec)\n",
            "INFO:tensorflow:global_step/sec: 70.0644\n",
            "INFO:tensorflow:loss = 32.064945, step = 1701 (1.427 sec)\n",
            "INFO:tensorflow:global_step/sec: 69.3297\n",
            "INFO:tensorflow:loss = 30.401932, step = 1801 (1.450 sec)\n",
            "INFO:tensorflow:global_step/sec: 72.1769\n",
            "INFO:tensorflow:loss = 29.174715, step = 1901 (1.377 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 2000 into /tmp/models/linear/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2020-01-28T23:24:54Z\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/linear/model.ckpt-2000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Finished evaluation at 2020-01-28-23:24:56\n",
            "INFO:tensorflow:Saving dict for global step 2000: accuracy = 0.8294, average_loss = 0.49750122, global_step = 2000, loss = 31.687975\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 2000: /tmp/models/linear/model.ckpt-2000\n",
            "INFO:tensorflow:global_step/sec: 25.769\n",
            "INFO:tensorflow:loss = 39.625103, step = 2001 (3.883 sec)\n",
            "INFO:tensorflow:global_step/sec: 73.4625\n",
            "INFO:tensorflow:loss = 29.722727, step = 2101 (1.364 sec)\n",
            "INFO:tensorflow:global_step/sec: 71.3886\n",
            "INFO:tensorflow:loss = 18.168343, step = 2201 (1.397 sec)\n",
            "INFO:tensorflow:global_step/sec: 70.8181\n",
            "INFO:tensorflow:loss = 20.410656, step = 2301 (1.411 sec)\n",
            "INFO:tensorflow:global_step/sec: 69.6612\n",
            "INFO:tensorflow:loss = 19.450668, step = 2401 (1.437 sec)\n",
            "INFO:tensorflow:global_step/sec: 72.1947\n",
            "INFO:tensorflow:loss = 40.91514, step = 2501 (1.387 sec)\n",
            "INFO:tensorflow:global_step/sec: 71.1518\n",
            "INFO:tensorflow:loss = 31.166111, step = 2601 (1.405 sec)\n",
            "INFO:tensorflow:global_step/sec: 70.163\n",
            "INFO:tensorflow:loss = 32.37959, step = 2701 (1.421 sec)\n",
            "INFO:tensorflow:global_step/sec: 71.6379\n",
            "INFO:tensorflow:loss = 22.445938, step = 2801 (1.396 sec)\n",
            "INFO:tensorflow:global_step/sec: 71.0854\n",
            "INFO:tensorflow:loss = 27.917372, step = 2901 (1.407 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 3000 into /tmp/models/linear/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2020-01-28T23:25:10Z\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/linear/model.ckpt-3000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Finished evaluation at 2020-01-28-23:25:12\n",
            "INFO:tensorflow:Saving dict for global step 3000: accuracy = 0.8388, average_loss = 0.47432587, global_step = 3000, loss = 30.21184\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 3000: /tmp/models/linear/model.ckpt-3000\n",
            "INFO:tensorflow:global_step/sec: 25.0352\n",
            "INFO:tensorflow:loss = 23.764828, step = 3001 (4.000 sec)\n",
            "INFO:tensorflow:global_step/sec: 70.0907\n",
            "INFO:tensorflow:loss = 34.317417, step = 3101 (1.424 sec)\n",
            "INFO:tensorflow:global_step/sec: 71.2348\n",
            "INFO:tensorflow:loss = 24.682533, step = 3201 (1.402 sec)\n",
            "INFO:tensorflow:global_step/sec: 75.6195\n",
            "INFO:tensorflow:loss = 18.028357, step = 3301 (1.325 sec)\n",
            "INFO:tensorflow:global_step/sec: 70.6113\n",
            "INFO:tensorflow:loss = 28.6536, step = 3401 (1.413 sec)\n",
            "INFO:tensorflow:global_step/sec: 70.8179\n",
            "INFO:tensorflow:loss = 35.568096, step = 3501 (1.414 sec)\n",
            "INFO:tensorflow:global_step/sec: 71.2977\n",
            "INFO:tensorflow:loss = 25.116405, step = 3601 (1.406 sec)\n",
            "INFO:tensorflow:global_step/sec: 72.7396\n",
            "INFO:tensorflow:loss = 33.921677, step = 3701 (1.373 sec)\n",
            "INFO:tensorflow:global_step/sec: 72.4945\n",
            "INFO:tensorflow:loss = 35.57686, step = 3801 (1.376 sec)\n",
            "INFO:tensorflow:global_step/sec: 72.1886\n",
            "INFO:tensorflow:loss = 21.0195, step = 3901 (1.389 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 4000 into /tmp/models/linear/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2020-01-28T23:25:27Z\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/linear/model.ckpt-4000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Finished evaluation at 2020-01-28-23:25:29\n",
            "INFO:tensorflow:Saving dict for global step 4000: accuracy = 0.8401, average_loss = 0.46905106, global_step = 4000, loss = 29.875864\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 4000: /tmp/models/linear/model.ckpt-4000\n",
            "INFO:tensorflow:global_step/sec: 25.6136\n",
            "INFO:tensorflow:loss = 26.495796, step = 4001 (3.900 sec)\n",
            "INFO:tensorflow:global_step/sec: 73.6551\n",
            "INFO:tensorflow:loss = 38.6, step = 4101 (1.362 sec)\n",
            "INFO:tensorflow:global_step/sec: 70.8532\n",
            "INFO:tensorflow:loss = 28.006618, step = 4201 (1.407 sec)\n",
            "INFO:tensorflow:global_step/sec: 73.5518\n",
            "INFO:tensorflow:loss = 27.710112, step = 4301 (1.363 sec)\n",
            "INFO:tensorflow:global_step/sec: 71.9281\n",
            "INFO:tensorflow:loss = 21.26177, step = 4401 (1.392 sec)\n",
            "INFO:tensorflow:global_step/sec: 72.9717\n",
            "INFO:tensorflow:loss = 22.310345, step = 4501 (1.369 sec)\n",
            "INFO:tensorflow:global_step/sec: 73.9845\n",
            "INFO:tensorflow:loss = 19.869438, step = 4601 (1.352 sec)\n",
            "INFO:tensorflow:global_step/sec: 73.3937\n",
            "INFO:tensorflow:loss = 10.47857, step = 4701 (1.364 sec)\n",
            "INFO:tensorflow:global_step/sec: 72.7757\n",
            "INFO:tensorflow:loss = 32.078213, step = 4801 (1.369 sec)\n",
            "INFO:tensorflow:global_step/sec: 71.2768\n",
            "INFO:tensorflow:loss = 20.28484, step = 4901 (1.402 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 5000 into /tmp/models/linear/model.ckpt.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/saver.py:963: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file APIs to delete files with this prefix.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2020-01-28T23:25:43Z\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/linear/model.ckpt-5000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Finished evaluation at 2020-01-28-23:25:45\n",
            "INFO:tensorflow:Saving dict for global step 5000: accuracy = 0.8414, average_loss = 0.4651065, global_step = 5000, loss = 29.624617\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 5000: /tmp/models/linear/model.ckpt-5000\n",
            "INFO:tensorflow:Loss for final step: 24.004433.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2020-01-28T23:25:45Z\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/linear/model.ckpt-5000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Finished evaluation at 2020-01-28-23:25:48\n",
            "INFO:tensorflow:Saving dict for global step 5000: accuracy = 0.8414, average_loss = 0.4651065, global_step = 5000, loss = 29.624617\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 5000: /tmp/models/linear/model.ckpt-5000\n",
            "Accuracy: 0.8414\n",
            "Loss: 0.4651065\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "a-1hE03c7_Yj"
      },
      "source": [
        "The linear model with default parameters achieves about **84.14% accuracy**.\n",
        "\n",
        "Let's see if we can do better with the `simple_dnn` AdaNet:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "9fAoRYd19eUs",
        "colab": {
          "test": {
            "output": "ignore",
            "timeout": 900
          },
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "921fa3ec-33a2-4799-f0e0-82c5e52e90b7"
      },
      "source": [
        "#@test {\"skip\": true}\n",
        "#@title Parameters\n",
        "LEARNING_RATE = 0.003  #@param {type:\"number\"}\n",
        "TRAIN_STEPS = 5000  #@param {type:\"integer\"}\n",
        "BATCH_SIZE = 64  #@param {type:\"integer\"}\n",
        "ADANET_ITERATIONS = 2  #@param {type:\"integer\"}\n",
        "\n",
        "estimator = adanet.Estimator(\n",
        "    head=head,\n",
        "    subnetwork_generator=simple_dnn.Generator(\n",
        "        feature_columns=feature_columns,\n",
        "        optimizer=tf.train.RMSPropOptimizer(learning_rate=LEARNING_RATE),\n",
        "        seed=RANDOM_SEED),\n",
        "    max_iteration_steps=TRAIN_STEPS // ADANET_ITERATIONS,\n",
        "    evaluator=adanet.Evaluator(\n",
        "        input_fn=input_fn(\"train\", training=False, batch_size=BATCH_SIZE),\n",
        "        steps=None),\n",
        "    config=make_config(\"simple_dnn\"))\n",
        "\n",
        "tf.estimator.train_and_evaluate(\n",
        "    estimator,\n",
        "    train_spec=tf.estimator.TrainSpec(\n",
        "        input_fn=input_fn(\"train\", training=True, batch_size=BATCH_SIZE),\n",
        "        max_steps=TRAIN_STEPS),\n",
        "    eval_spec=tf.estimator.EvalSpec(\n",
        "        input_fn=input_fn(\"test\", training=False, batch_size=BATCH_SIZE),\n",
        "        steps=None,\n",
        "        start_delay_secs=1,\n",
        "        throttle_secs=1,  \n",
        "    ))\n",
        "\n",
        "results = estimator.evaluate(\n",
        "    input_fn(\"test\", training=False, batch_size=BATCH_SIZE),\n",
        "    steps=None)\n",
        "print(\"Accuracy:\", results[\"accuracy\"])\n",
        "print(\"Loss:\", results[\"average_loss\"])"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_dnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b0cfd1a90>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
            "INFO:tensorflow:Not using Distribute Coordinator.\n",
            "INFO:tensorflow:Running training and evaluation locally (non-distributed).\n",
            "INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 1000 or save_checkpoints_secs None.\n",
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_dnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b0cfd1e48>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_dnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b0cfd1e48>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:\n",
            "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
            "For more information, please see:\n",
            "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
            "  * https://github.com/tensorflow/addons\n",
            "  * https://github.com/tensorflow/io (for I/O related ops)\n",
            "If you depend on functionality not listed there, please file an issue.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:\n",
            "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
            "For more information, please see:\n",
            "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
            "  * https://github.com/tensorflow/addons\n",
            "  * https://github.com/tensorflow/io (for I/O related ops)\n",
            "If you depend on functionality not listed there, please file an issue.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/feature_column/feature_column.py:206: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/feature_column/feature_column.py:206: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/feature_column/feature_column.py:2158: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/feature_column/feature_column.py:2158: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/feature_column/feature_column.py:207: NumericColumn._variable_shape (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/feature_column/feature_column.py:207: NumericColumn._variable_shape (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/adanet/examples/simple_dnn.py:91: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use keras.layers.Dense instead.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/adanet/examples/simple_dnn.py:91: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use keras.layers.Dense instead.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/layers/core.py:187: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use `layer.__call__` method instead.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/layers/core.py:187: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use `layer.__call__` method instead.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/adanet/examples/simple_dnn.py:86: dropout (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use keras.layers.dropout instead.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/adanet/examples/simple_dnn.py:86: dropout (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use keras.layers.dropout instead.\n",
            "WARNING:absl:The `build_mixture_weights_train_op` method is deprecated. Please use the `Ensembler#build_train_op` instead.\n",
            "WARNING:absl:The `build_mixture_weights_train_op` method is deprecated. Please use the `Ensembler#build_train_op` instead.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Create CheckpointSaverHook.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Create CheckpointSaverHook.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 0 into /tmp/models/simple_dnn/model.ckpt.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 0 into /tmp/models/simple_dnn/model.ckpt.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 2.603788, step = 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 2.603788, step = 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 58.0022\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 58.0022\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.8165605, step = 100 (1.741 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.8165605, step = 100 (1.741 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 68.515\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 68.515\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.66095746, step = 200 (1.446 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.66095746, step = 200 (1.446 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 68.2362\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 68.2362\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.38084134, step = 300 (1.470 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.38084134, step = 300 (1.470 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 68.5841\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 68.5841\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.49972004, step = 400 (1.452 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.49972004, step = 400 (1.452 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.1803\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.1803\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.4046151, step = 500 (1.511 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.4046151, step = 500 (1.511 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 68.5868\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 68.5868\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.8298083, step = 600 (1.461 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.8298083, step = 600 (1.461 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.7415\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.7415\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.47103176, step = 700 (1.496 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.47103176, step = 700 (1.496 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 68.2163\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 68.2163\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.6030396, step = 800 (1.469 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.6030396, step = 800 (1.469 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 68.0853\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 68.0853\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.2797109, step = 900 (1.465 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.2797109, step = 900 (1.465 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 1000 into /tmp/models/simple_dnn/model.ckpt.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 1000 into /tmp/models/simple_dnn/model.ckpt.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_dnn/model.ckpt-1000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_dnn/model.ckpt-1000\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/adanet/core/estimator.py:1408: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "To construct input pipelines, use the `tf.data` module.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/adanet/core/estimator.py:1408: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "To construct input pipelines, use the `tf.data` module.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:`tf.train.start_queue_runners()` was called when no queue runners were defined. You can safely remove the call to this deprecated function.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:`tf.train.start_queue_runners()` was called when no queue runners were defined. You can safely remove the call to this deprecated function.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_dnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b041ddb70>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_dnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b041ddb70>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Starting evaluation at 2020-01-28T23:26:21Z\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Starting evaluation at 2020-01-28T23:26:21Z\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_dnn/model.ckpt-1000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_dnn/model.ckpt-1000\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Finished evaluation at 2020-01-28-23:26:25\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Finished evaluation at 2020-01-28-23:26:25\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.8222, average_loss = 0.50020385, best_ensemble_index_0 = 1, global_step = 1000, iteration = 0, loss = 0.4996074\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.8222, average_loss = 0.50020385, best_ensemble_index_0 = 1, global_step = 1000, iteration = 0, loss = 0.4996074\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmp/models/simple_dnn/model.ckpt-1000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmp/models/simple_dnn/model.ckpt-1000\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 4.73282\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 4.73282\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.4469478, step = 1000 (21.128 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.4469478, step = 1000 (21.128 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 69.1939\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 69.1939\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.49168974, step = 1100 (1.453 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.49168974, step = 1100 (1.453 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.7079\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.7079\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.59497625, step = 1200 (1.493 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.59497625, step = 1200 (1.493 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 68.1564\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 68.1564\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.35278708, step = 1300 (1.472 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.35278708, step = 1300 (1.472 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 65.3813\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 65.3813\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.50315446, step = 1400 (1.530 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.50315446, step = 1400 (1.530 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.0373\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.0373\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.3651929, step = 1500 (1.507 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.3651929, step = 1500 (1.507 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.993\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.993\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.25101003, step = 1600 (1.497 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.25101003, step = 1600 (1.497 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 67.1637\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 67.1637\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.493657, step = 1700 (1.484 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.493657, step = 1700 (1.484 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 69.4439\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 69.4439\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.40216514, step = 1800 (1.440 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.40216514, step = 1800 (1.440 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 68.2726\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 68.2726\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.41076538, step = 1900 (1.464 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.41076538, step = 1900 (1.464 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 2000 into /tmp/models/simple_dnn/model.ckpt.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 2000 into /tmp/models/simple_dnn/model.ckpt.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_dnn/model.ckpt-2000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_dnn/model.ckpt-2000\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:`tf.train.start_queue_runners()` was called when no queue runners were defined. You can safely remove the call to this deprecated function.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:`tf.train.start_queue_runners()` was called when no queue runners were defined. You can safely remove the call to this deprecated function.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_dnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b04009e48>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_dnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b04009e48>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Starting evaluation at 2020-01-28T23:26:55Z\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Starting evaluation at 2020-01-28T23:26:55Z\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_dnn/model.ckpt-2000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_dnn/model.ckpt-2000\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Finished evaluation at 2020-01-28-23:26:58\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Finished evaluation at 2020-01-28-23:26:58\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving dict for global step 2000: accuracy = 0.8533, average_loss = 0.41756538, best_ensemble_index_0 = 1, global_step = 2000, iteration = 0, loss = 0.41665503\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving dict for global step 2000: accuracy = 0.8533, average_loss = 0.41756538, best_ensemble_index_0 = 1, global_step = 2000, iteration = 0, loss = 0.41665503\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 2000: /tmp/models/simple_dnn/model.ckpt-2000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 2000: /tmp/models/simple_dnn/model.ckpt-2000\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 5.11941\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 5.11941\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.48366874, step = 2000 (19.533 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.48366874, step = 2000 (19.533 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 67.4132\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 67.4132\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.4200433, step = 2100 (1.485 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.4200433, step = 2100 (1.485 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 64.9966\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 64.9966\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.2366377, step = 2200 (1.543 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.2366377, step = 2200 (1.543 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.8132\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.8132\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.25099033, step = 2300 (1.494 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.25099033, step = 2300 (1.494 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 67.8192\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 67.8192\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.2939929, step = 2400 (1.480 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.2939929, step = 2400 (1.480 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 2500 into /tmp/models/simple_dnn/model.ckpt.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 2500 into /tmp/models/simple_dnn/model.ckpt.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_dnn/model.ckpt-2500\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_dnn/model.ckpt-2500\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:`tf.train.start_queue_runners()` was called when no queue runners were defined. You can safely remove the call to this deprecated function.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:`tf.train.start_queue_runners()` was called when no queue runners were defined. You can safely remove the call to this deprecated function.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_dnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b03125828>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_dnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b03125828>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Starting evaluation at 2020-01-28T23:27:21Z\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Starting evaluation at 2020-01-28T23:27:21Z\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_dnn/model.ckpt-2500\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_dnn/model.ckpt-2500\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Finished evaluation at 2020-01-28-23:27:24\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Finished evaluation at 2020-01-28-23:27:24\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving dict for global step 2500: accuracy = 0.8506, average_loss = 0.42281744, best_ensemble_index_0 = 1, global_step = 2500, iteration = 0, loss = 0.42212486\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving dict for global step 2500: accuracy = 0.8506, average_loss = 0.42281744, best_ensemble_index_0 = 1, global_step = 2500, iteration = 0, loss = 0.42212486\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 2500: /tmp/models/simple_dnn/model.ckpt-2500\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 2500: /tmp/models/simple_dnn/model.ckpt-2500\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Loss for final step: 0.30081838.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Loss for final step: 0.30081838.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_dnn/model.ckpt-2500\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_dnn/model.ckpt-2500\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:`tf.train.start_queue_runners()` was called when no queue runners were defined. You can safely remove the call to this deprecated function.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:`tf.train.start_queue_runners()` was called when no queue runners were defined. You can safely remove the call to this deprecated function.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_dnn/temp_model_dir/1580254058504', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b0c97a278>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_dnn/temp_model_dir/1580254058504', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b0c97a278>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n",
            "WARNING:absl:The `build_mixture_weights_train_op` method is deprecated. Please use the `Ensembler#build_train_op` instead.\n",
            "WARNING:absl:The `build_mixture_weights_train_op` method is deprecated. Please use the `Ensembler#build_train_op` instead.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Create CheckpointSaverHook.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Create CheckpointSaverHook.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/saver.py:1069: get_checkpoint_mtimes (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file utilities to get mtimes.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/saver.py:1069: get_checkpoint_mtimes (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file utilities to get mtimes.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 0 into /tmp/models/simple_dnn/temp_model_dir/1580254058504/model.ckpt.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 0 into /tmp/models/simple_dnn/temp_model_dir/1580254058504/model.ckpt.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_dnn/model.ckpt-2500\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_dnn/model.ckpt-2500\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 2501 into /tmp/models/simple_dnn/temp_model_dir/1580254058504/model.ckpt.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 2501 into /tmp/models/simple_dnn/temp_model_dir/1580254058504/model.ckpt.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.31168938, step = 2501\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.31168938, step = 2501\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Loss for final step: 0.31168938.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Loss for final step: 0.31168938.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_dnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b05bfd978>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_dnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b05bfd978>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n",
            "WARNING:absl:The `build_mixture_weights_train_op` method is deprecated. Please use the `Ensembler#build_train_op` instead.\n",
            "WARNING:absl:The `build_mixture_weights_train_op` method is deprecated. Please use the `Ensembler#build_train_op` instead.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Create CheckpointSaverHook.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Create CheckpointSaverHook.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_dnn/increment.ckpt-1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_dnn/increment.ckpt-1\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 2500 into /tmp/models/simple_dnn/model.ckpt.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 2500 into /tmp/models/simple_dnn/model.ckpt.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.31168938, step = 2500\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.31168938, step = 2500\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 54.6787\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 54.6787\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.17422614, step = 2600 (1.841 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.17422614, step = 2600 (1.841 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.6588\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.6588\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.5036852, step = 2700 (1.495 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.5036852, step = 2700 (1.495 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 65.1429\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 65.1429\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.27954224, step = 2800 (1.539 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.27954224, step = 2800 (1.539 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 69.1591\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 69.1591\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.39685076, step = 2900 (1.443 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.39685076, step = 2900 (1.443 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 68.5735\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 68.5735\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.27495292, step = 3000 (1.454 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.27495292, step = 3000 (1.454 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 65.6816\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 65.6816\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.50383085, step = 3100 (1.523 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.50383085, step = 3100 (1.523 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 67.5642\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 67.5642\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.44185284, step = 3200 (1.483 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.44185284, step = 3200 (1.483 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 65.5454\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 65.5454\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.450566, step = 3300 (1.525 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.450566, step = 3300 (1.525 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.2342\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.2342\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.2405195, step = 3400 (1.511 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.2405195, step = 3400 (1.511 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 3500 into /tmp/models/simple_dnn/model.ckpt.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 3500 into /tmp/models/simple_dnn/model.ckpt.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 47.5938\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 47.5938\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.38556522, step = 3500 (2.099 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.38556522, step = 3500 (2.099 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 69.4946\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 69.4946\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.34398308, step = 3600 (1.437 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.34398308, step = 3600 (1.437 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 67.633\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 67.633\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.6122652, step = 3700 (1.481 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.6122652, step = 3700 (1.481 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 64.7956\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 64.7956\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.2923677, step = 3800 (1.544 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.2923677, step = 3800 (1.544 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 67.1339\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 67.1339\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.43072838, step = 3900 (1.491 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.43072838, step = 3900 (1.491 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.2459\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.2459\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.38637918, step = 4000 (1.509 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.38637918, step = 4000 (1.509 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 64.2186\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 64.2186\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.19899742, step = 4100 (1.564 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.19899742, step = 4100 (1.564 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 65.458\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 65.458\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.3459694, step = 4200 (1.520 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.3459694, step = 4200 (1.520 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 65.7555\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 65.7555\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.41656035, step = 4300 (1.531 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.41656035, step = 4300 (1.531 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 64.6976\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 64.6976\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.34805182, step = 4400 (1.535 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.34805182, step = 4400 (1.535 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 4500 into /tmp/models/simple_dnn/model.ckpt.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 4500 into /tmp/models/simple_dnn/model.ckpt.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 58.2602\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 58.2602\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.42429283, step = 4500 (1.711 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.42429283, step = 4500 (1.711 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.7339\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.7339\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.37127605, step = 4600 (1.502 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.37127605, step = 4600 (1.502 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.3439\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.3439\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.20735526, step = 4700 (1.505 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.20735526, step = 4700 (1.505 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.7549\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.7549\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.19874632, step = 4800 (1.503 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.19874632, step = 4800 (1.503 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.6383\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 66.6383\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.18172987, step = 4900 (1.502 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.18172987, step = 4900 (1.502 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 5000 into /tmp/models/simple_dnn/model.ckpt.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 5000 into /tmp/models/simple_dnn/model.ckpt.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Loss for final step: 0.20381317.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Loss for final step: 0.20381317.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_dnn/model.ckpt-5000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_dnn/model.ckpt-5000\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:`tf.train.start_queue_runners()` was called when no queue runners were defined. You can safely remove the call to this deprecated function.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:`tf.train.start_queue_runners()` was called when no queue runners were defined. You can safely remove the call to this deprecated function.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_dnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b03b80e80>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_dnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b03b80e80>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Starting evaluation at 2020-01-28T23:28:43Z\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Starting evaluation at 2020-01-28T23:28:43Z\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_dnn/model.ckpt-5000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_dnn/model.ckpt-5000\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Finished evaluation at 2020-01-28-23:28:47\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Finished evaluation at 2020-01-28-23:28:47\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving dict for global step 5000: accuracy = 0.8564, average_loss = 0.40214804, best_ensemble_index_0 = 1, best_ensemble_index_1 = 2, global_step = 5000, iteration = 1, loss = 0.4015974\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving dict for global step 5000: accuracy = 0.8564, average_loss = 0.40214804, best_ensemble_index_0 = 1, best_ensemble_index_1 = 2, global_step = 5000, iteration = 1, loss = 0.4015974\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 5000: /tmp/models/simple_dnn/model.ckpt-5000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 5000: /tmp/models/simple_dnn/model.ckpt-5000\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Accuracy: 0.8564\n",
            "Loss: 0.40214804\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ysWsJ3zXDwNx"
      },
      "source": [
        "The `simple_dnn` AdaNet model with default parameters achieves about **85.64%\n",
        "accuracy**.\n",
        "\n",
        "This improvement can be attributed to `simple_dnn` searching over\n",
        "fully-connected neural networks which have more expressive power than the linear\n",
        "model due to their non-linear activations.\n",
        "\n",
        "Fully-connected layers are permutation invariant to their inputs, meaning that\n",
        "if we consistently swapped two pixels before training, the final model would\n",
        "perform identically. However, there is spatial and locality information in\n",
        "images that we should try to capture. Applying a few convolutions to our inputs\n",
        "will allow us to do so, and that will require defining a custom\n",
        "`adanet.subnetwork.Builder` and `adanet.subnetwork.Generator`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "D3IE6-9vFVlg"
      },
      "source": [
        "## Define a convolutional AdaNet model\n",
        "\n",
        "Creating a new search space for AdaNet to explore is straightforward. There are\n",
        "two abstract classes you need to extend:\n",
        "\n",
        "1.  `adanet.subnetwork.Builder`\n",
        "2.  `adanet.subnetwork.Generator`\n",
        "\n",
        "Similar to the tf.estimator.Estimator `model_fn`, `adanet.subnetwork.Builder`\n",
        "allows you to define your own TensorFlow graph for creating a neural network,\n",
        "and specify the training operations.\n",
        "\n",
        "Below we define one that applies a 2D convolution, max-pooling, and then a\n",
        "fully-connected layer to the images:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "IsYJ97tRwBkt",
        "colab": {
          "test": {
            "output": "ignore",
            "timeout": 900
          }
        }
      },
      "source": [
        "class SimpleCNNBuilder(adanet.subnetwork.Builder):\n",
        "  \"\"\"Builds a CNN subnetwork for AdaNet.\"\"\"\n",
        "\n",
        "  def __init__(self, learning_rate, max_iteration_steps, seed):\n",
        "    \"\"\"Initializes a `SimpleCNNBuilder`.\n",
        "\n",
        "    Args:\n",
        "      learning_rate: The float learning rate to use.\n",
        "      max_iteration_steps: The number of steps per iteration.\n",
        "      seed: The random seed.\n",
        "\n",
        "    Returns:\n",
        "      An instance of `SimpleCNNBuilder`.\n",
        "    \"\"\"\n",
        "    self._learning_rate = learning_rate\n",
        "    self._max_iteration_steps = max_iteration_steps\n",
        "    self._seed = seed\n",
        "\n",
        "  def build_subnetwork(self,\n",
        "                       features,\n",
        "                       logits_dimension,\n",
        "                       training,\n",
        "                       iteration_step,\n",
        "                       summary,\n",
        "                       previous_ensemble=None):\n",
        "    \"\"\"See `adanet.subnetwork.Builder`.\"\"\"\n",
        "    images = list(features.values())[0]\n",
        "    \n",
        "    # Visualize some of the input images in TensorBoard.\n",
        "    summary.image(\"images\", images)\n",
        "    \n",
        "    kernel_initializer = tf.keras.initializers.he_normal(seed=self._seed)\n",
        "    x = tf.keras.layers.Conv2D(\n",
        "        filters=16,\n",
        "        kernel_size=3,\n",
        "        padding=\"same\",\n",
        "        activation=\"relu\",\n",
        "        kernel_initializer=kernel_initializer)(\n",
        "            images)\n",
        "    x = tf.keras.layers.MaxPool2D(pool_size=2, strides=2)(x)\n",
        "    x = tf.keras.layers.Flatten()(x)\n",
        "    x = tf.keras.layers.Dense(\n",
        "        units=64, activation=\"relu\", kernel_initializer=kernel_initializer)(\n",
        "            x)\n",
        "\n",
        "    # The `Head` passed to adanet.Estimator will apply the softmax activation.\n",
        "    logits = tf.keras.layers.Dense(\n",
        "        units=10, activation=None, kernel_initializer=kernel_initializer)(\n",
        "            x)\n",
        "\n",
        "    # Use a constant complexity measure, since all subnetworks have the same\n",
        "    # architecture and hyperparameters.\n",
        "    complexity = tf.constant(1)\n",
        "\n",
        "    return adanet.Subnetwork(\n",
        "        last_layer=x,\n",
        "        logits=logits,\n",
        "        complexity=complexity,\n",
        "        persisted_tensors={})\n",
        "\n",
        "  def build_subnetwork_train_op(self,\n",
        "                                subnetwork,\n",
        "                                loss,\n",
        "                                var_list,\n",
        "                                labels,\n",
        "                                iteration_step,\n",
        "                                summary,\n",
        "                                previous_ensemble=None):\n",
        "    \"\"\"See `adanet.subnetwork.Builder`.\"\"\"\n",
        "\n",
        "    # Momentum optimizer with cosine learning rate decay works well with CNNs.\n",
        "    learning_rate = tf.train.cosine_decay(\n",
        "        learning_rate=self._learning_rate,\n",
        "        global_step=iteration_step,\n",
        "        decay_steps=self._max_iteration_steps)\n",
        "    optimizer = tf.train.MomentumOptimizer(learning_rate, .9)\n",
        "    # NOTE: The `adanet.Estimator` increments the global step.\n",
        "    return optimizer.minimize(loss=loss, var_list=var_list)\n",
        "\n",
        "  def build_mixture_weights_train_op(self, loss, var_list, logits, labels,\n",
        "                                     iteration_step, summary):\n",
        "    \"\"\"See `adanet.subnetwork.Builder`.\"\"\"\n",
        "    return tf.no_op(\"mixture_weights_train_op\")\n",
        "\n",
        "  @property\n",
        "  def name(self):\n",
        "    \"\"\"See `adanet.subnetwork.Builder`.\"\"\"\n",
        "    return \"simple_cnn\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "OFamPrZHJ5ii"
      },
      "source": [
        "Next, we extend a `adanet.subnetwork.Generator`, which defines the search\n",
        "space of candidate `SimpleCNNBuilders` to consider including the final network.\n",
        "It can create one or more at each iteration with different parameters, and the\n",
        "AdaNet algorithm will select the candidate that best improves the overall neural\n",
        "network's `adanet_loss` on the training set.\n",
        "\n",
        "The one below is very simple: it always creates the same architecture, but gives\n",
        "it a different random seed at each iteration:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "-BAnb_XGwhRy",
        "colab": {
          "test": {
            "output": "ignore",
            "timeout": 900
          }
        }
      },
      "source": [
        "class SimpleCNNGenerator(adanet.subnetwork.Generator):\n",
        "  \"\"\"Generates a `SimpleCNN` at each iteration.\n",
        "  \"\"\"\n",
        "\n",
        "  def __init__(self, learning_rate, max_iteration_steps, seed=None):\n",
        "    \"\"\"Initializes a `Generator` that builds `SimpleCNNs`.\n",
        "\n",
        "    Args:\n",
        "      learning_rate: The float learning rate to use.\n",
        "      max_iteration_steps: The number of steps per iteration.\n",
        "      seed: The random seed.\n",
        "\n",
        "    Returns:\n",
        "      An instance of `Generator`.\n",
        "    \"\"\"\n",
        "    self._seed = seed\n",
        "    self._dnn_builder_fn = functools.partial(\n",
        "        SimpleCNNBuilder,\n",
        "        learning_rate=learning_rate,\n",
        "        max_iteration_steps=max_iteration_steps)\n",
        "\n",
        "  def generate_candidates(self, previous_ensemble, iteration_number,\n",
        "                          previous_ensemble_reports, all_reports):\n",
        "    \"\"\"See `adanet.subnetwork.Generator`.\"\"\"\n",
        "    seed = self._seed\n",
        "    # Change the seed according to the iteration so that each subnetwork\n",
        "    # learns something different.\n",
        "    if seed is not None:\n",
        "      seed += iteration_number\n",
        "    return [self._dnn_builder_fn(seed=seed)]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "8sdvharsLJ1T"
      },
      "source": [
        "With these defined, we pass them into a new `adanet.Estimator`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "-Fhi1SjkzVBt",
        "colab": {
          "test": {
            "output": "ignore",
            "timeout": 900
          },
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "66cd7176-7f06-42fb-ed3f-7126eb3e7df6"
      },
      "source": [
        "#@title Parameters\n",
        "LEARNING_RATE = 0.05  #@param {type:\"number\"}\n",
        "TRAIN_STEPS = 5000  #@param {type:\"integer\"}\n",
        "BATCH_SIZE = 64  #@param {type:\"integer\"}\n",
        "ADANET_ITERATIONS = 2  #@param {type:\"integer\"}\n",
        "\n",
        "max_iteration_steps = TRAIN_STEPS // ADANET_ITERATIONS\n",
        "estimator = adanet.Estimator(\n",
        "    head=head,\n",
        "    subnetwork_generator=SimpleCNNGenerator(\n",
        "        learning_rate=LEARNING_RATE,\n",
        "        max_iteration_steps=max_iteration_steps,\n",
        "        seed=RANDOM_SEED),\n",
        "    max_iteration_steps=max_iteration_steps,\n",
        "    evaluator=adanet.Evaluator(\n",
        "        input_fn=input_fn(\"train\", training=False, batch_size=BATCH_SIZE),\n",
        "        steps=None),\n",
        "    adanet_loss_decay=.99,\n",
        "    config=make_config(\"simple_cnn\"))\n",
        "\n",
        "results, _ = tf.estimator.train_and_evaluate(\n",
        "    estimator,\n",
        "    train_spec=tf.estimator.TrainSpec(\n",
        "        input_fn=input_fn(\"train\", training=True, batch_size=BATCH_SIZE),\n",
        "        max_steps=TRAIN_STEPS),\n",
        "    eval_spec=tf.estimator.EvalSpec(\n",
        "        input_fn=input_fn(\"test\", training=False, batch_size=BATCH_SIZE),\n",
        "        steps=None,\n",
        "        start_delay_secs=1,\n",
        "        throttle_secs=1,  \n",
        "    ))\n",
        "print(\"Accuracy:\", results[\"accuracy\"])\n",
        "print(\"Loss:\", results[\"average_loss\"])"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_cnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b04fad668>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_cnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b04fad668>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Not using Distribute Coordinator.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Not using Distribute Coordinator.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running training and evaluation locally (non-distributed).\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running training and evaluation locally (non-distributed).\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 1000 or save_checkpoints_secs None.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 1000 or save_checkpoints_secs None.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_cnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b04fadc88>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_cnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b04fadc88>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From <ipython-input-10-13ffa07860f9>:59: calling Subnetwork.__new__ (from adanet.subnetwork.generator) with persisted_tensors is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "`persisted_tensors` is deprecated, please use `shared` instead.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From <ipython-input-10-13ffa07860f9>:59: calling Subnetwork.__new__ (from adanet.subnetwork.generator) with persisted_tensors is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "`persisted_tensors` is deprecated, please use `shared` instead.\n",
            "WARNING:absl:The `build_mixture_weights_train_op` method is deprecated. Please use the `Ensembler#build_train_op` instead.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Create CheckpointSaverHook.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Create CheckpointSaverHook.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 0 into /tmp/models/simple_cnn/model.ckpt.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 0 into /tmp/models/simple_cnn/model.ckpt.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 2.7829337, step = 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 2.7829337, step = 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 34.8672\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 34.8672\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.37638825, step = 100 (2.870 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.37638825, step = 100 (2.870 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 35.4227\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 35.4227\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.5209242, step = 200 (2.824 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.5209242, step = 200 (2.824 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 36.8113\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 36.8113\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.27607995, step = 300 (2.716 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.27607995, step = 300 (2.716 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 35.5856\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 35.5856\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.3605134, step = 400 (2.810 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.3605134, step = 400 (2.810 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 37.8285\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 37.8285\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.2832445, step = 500 (2.646 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.2832445, step = 500 (2.646 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 37.0919\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 37.0919\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.40479934, step = 600 (2.693 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.40479934, step = 600 (2.693 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 36.8642\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 36.8642\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.43839163, step = 700 (2.715 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.43839163, step = 700 (2.715 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 37.4738\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 37.4738\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.37346122, step = 800 (2.666 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.37346122, step = 800 (2.666 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 36.9895\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 36.9895\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.12023185, step = 900 (2.705 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.12023185, step = 900 (2.705 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 1000 into /tmp/models/simple_cnn/model.ckpt.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 1000 into /tmp/models/simple_cnn/model.ckpt.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_cnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b04d81a20>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_cnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b04d81a20>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Starting evaluation at 2020-01-28T23:29:18Z\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Starting evaluation at 2020-01-28T23:29:18Z\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_cnn/model.ckpt-1000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_cnn/model.ckpt-1000\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Finished evaluation at 2020-01-28-23:29:21\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Finished evaluation at 2020-01-28-23:29:21\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.8816, average_loss = 0.33634645, best_ensemble_index_0 = 0, global_step = 1000, iteration = 0, loss = 0.3361071\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.8816, average_loss = 0.33634645, best_ensemble_index_0 = 0, global_step = 1000, iteration = 0, loss = 0.3361071\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmp/models/simple_cnn/model.ckpt-1000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: /tmp/models/simple_cnn/model.ckpt-1000\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 12.6757\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 12.6757\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.33306035, step = 1000 (7.889 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.33306035, step = 1000 (7.889 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 36.7901\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 36.7901\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.26332143, step = 1100 (2.718 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.26332143, step = 1100 (2.718 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 38.053\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 38.053\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.5548452, step = 1200 (2.628 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.5548452, step = 1200 (2.628 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 37.1053\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 37.1053\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.3751756, step = 1300 (2.695 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.3751756, step = 1300 (2.695 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 38.2062\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 38.2062\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.40759182, step = 1400 (2.617 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.40759182, step = 1400 (2.617 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 35.6691\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 35.6691\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.43741018, step = 1500 (2.809 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.43741018, step = 1500 (2.809 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 36.6863\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 36.6863\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.11292698, step = 1600 (2.722 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.11292698, step = 1600 (2.722 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 36.7414\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 36.7414\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.28705102, step = 1700 (2.722 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.28705102, step = 1700 (2.722 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 37.8563\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 37.8563\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.38228264, step = 1800 (2.644 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.38228264, step = 1800 (2.644 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 37.3185\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 37.3185\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.20412554, step = 1900 (2.675 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.20412554, step = 1900 (2.675 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 2000 into /tmp/models/simple_cnn/model.ckpt.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 2000 into /tmp/models/simple_cnn/model.ckpt.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_cnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b031c36a0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_cnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b031c36a0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Starting evaluation at 2020-01-28T23:29:50Z\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Starting evaluation at 2020-01-28T23:29:50Z\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_cnn/model.ckpt-2000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_cnn/model.ckpt-2000\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Finished evaluation at 2020-01-28-23:29:54\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Finished evaluation at 2020-01-28-23:29:54\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving dict for global step 2000: accuracy = 0.8943, average_loss = 0.2866584, best_ensemble_index_0 = 0, global_step = 2000, iteration = 0, loss = 0.28647542\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving dict for global step 2000: accuracy = 0.8943, average_loss = 0.2866584, best_ensemble_index_0 = 0, global_step = 2000, iteration = 0, loss = 0.28647542\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 2000: /tmp/models/simple_cnn/model.ckpt-2000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 2000: /tmp/models/simple_cnn/model.ckpt-2000\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 12.5854\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 12.5854\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.25820497, step = 2000 (7.947 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.25820497, step = 2000 (7.947 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 36.7172\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 36.7172\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.19789831, step = 2100 (2.723 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.19789831, step = 2100 (2.723 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 37.8414\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 37.8414\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.1775988, step = 2200 (2.642 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.1775988, step = 2200 (2.642 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 35.7965\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 35.7965\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.12481536, step = 2300 (2.797 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.12481536, step = 2300 (2.797 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 37.1199\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 37.1199\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.11573415, step = 2400 (2.691 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.11573415, step = 2400 (2.691 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 2500 into /tmp/models/simple_cnn/model.ckpt.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 2500 into /tmp/models/simple_cnn/model.ckpt.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_cnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b0e8994a8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_cnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b0e8994a8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Starting evaluation at 2020-01-28T23:30:08Z\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Starting evaluation at 2020-01-28T23:30:08Z\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_cnn/model.ckpt-2500\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_cnn/model.ckpt-2500\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Finished evaluation at 2020-01-28-23:30:12\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Finished evaluation at 2020-01-28-23:30:12\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving dict for global step 2500: accuracy = 0.9021, average_loss = 0.273015, best_ensemble_index_0 = 0, global_step = 2500, iteration = 0, loss = 0.27311632\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving dict for global step 2500: accuracy = 0.9021, average_loss = 0.273015, best_ensemble_index_0 = 0, global_step = 2500, iteration = 0, loss = 0.27311632\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 2500: /tmp/models/simple_cnn/model.ckpt-2500\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 2500: /tmp/models/simple_cnn/model.ckpt-2500\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Loss for final step: 0.14896837.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Loss for final step: 0.14896837.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_cnn/temp_model_dir/1580254213075', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b03afe4a8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_cnn/temp_model_dir/1580254213075', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b03afe4a8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n",
            "WARNING:absl:The `build_mixture_weights_train_op` method is deprecated. Please use the `Ensembler#build_train_op` instead.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Create CheckpointSaverHook.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Create CheckpointSaverHook.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 0 into /tmp/models/simple_cnn/temp_model_dir/1580254213075/model.ckpt.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 0 into /tmp/models/simple_cnn/temp_model_dir/1580254213075/model.ckpt.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_cnn/model.ckpt-2500\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_cnn/model.ckpt-2500\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 2501 into /tmp/models/simple_cnn/temp_model_dir/1580254213075/model.ckpt.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 2501 into /tmp/models/simple_cnn/temp_model_dir/1580254213075/model.ckpt.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.12337765, step = 2501\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.12337765, step = 2501\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Loss for final step: 0.12337765.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Loss for final step: 0.12337765.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_cnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b0cfdb8d0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_cnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b0cfdb8d0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n",
            "WARNING:absl:The `build_mixture_weights_train_op` method is deprecated. Please use the `Ensembler#build_train_op` instead.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Create CheckpointSaverHook.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Create CheckpointSaverHook.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_cnn/increment.ckpt-1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_cnn/increment.ckpt-1\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 2500 into /tmp/models/simple_cnn/model.ckpt.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 2500 into /tmp/models/simple_cnn/model.ckpt.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.12337765, step = 2500\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.12337765, step = 2500\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 26.3029\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 26.3029\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.12534487, step = 2600 (3.806 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.12534487, step = 2600 (3.806 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 28.0782\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 28.0782\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.34530002, step = 2700 (3.563 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.34530002, step = 2700 (3.563 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 28.9404\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 28.9404\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.10920033, step = 2800 (3.451 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.10920033, step = 2800 (3.451 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 28.3223\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 28.3223\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.23371549, step = 2900 (3.533 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.23371549, step = 2900 (3.533 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 29.1425\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 29.1425\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.128768, step = 3000 (3.436 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.128768, step = 3000 (3.436 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 29.324\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 29.324\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.2285847, step = 3100 (3.407 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.2285847, step = 3100 (3.407 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 28.3976\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 28.3976\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.24578372, step = 3200 (3.523 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.24578372, step = 3200 (3.523 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 28.8632\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 28.8632\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.2148009, step = 3300 (3.463 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.2148009, step = 3300 (3.463 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 29.0824\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 29.0824\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.0779793, step = 3400 (3.437 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.0779793, step = 3400 (3.437 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 3500 into /tmp/models/simple_cnn/model.ckpt.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 3500 into /tmp/models/simple_cnn/model.ckpt.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 25.7493\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 25.7493\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.2121433, step = 3500 (3.885 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.2121433, step = 3500 (3.885 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 29.1679\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 29.1679\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.15474264, step = 3600 (3.425 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.15474264, step = 3600 (3.425 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 29.2246\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 29.2246\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.4731341, step = 3700 (3.422 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.4731341, step = 3700 (3.422 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 28.7829\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 28.7829\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.35375696, step = 3800 (3.476 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.35375696, step = 3800 (3.476 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 28.8149\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 28.8149\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.30882052, step = 3900 (3.469 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.30882052, step = 3900 (3.469 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 29.3919\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 29.3919\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.3272449, step = 4000 (3.405 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.3272449, step = 4000 (3.405 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 28.3549\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 28.3549\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.07785181, step = 4100 (3.530 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.07785181, step = 4100 (3.530 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 28.6643\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 28.6643\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.2796491, step = 4200 (3.485 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.2796491, step = 4200 (3.485 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 29.1041\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 29.1041\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.3621142, step = 4300 (3.434 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.3621142, step = 4300 (3.434 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 28.7995\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 28.7995\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.16525124, step = 4400 (3.477 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.16525124, step = 4400 (3.477 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 4500 into /tmp/models/simple_cnn/model.ckpt.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 4500 into /tmp/models/simple_cnn/model.ckpt.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 27.6201\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 27.6201\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.259004, step = 4500 (3.616 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.259004, step = 4500 (3.616 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 29.673\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 29.673\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.19452739, step = 4600 (3.374 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.19452739, step = 4600 (3.374 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 28.3729\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 28.3729\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.15597239, step = 4700 (3.521 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.15597239, step = 4700 (3.521 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 28.9884\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 28.9884\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.13326406, step = 4800 (3.452 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.13326406, step = 4800 (3.452 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 29.2913\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 29.2913\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.115908034, step = 4900 (3.413 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.115908034, step = 4900 (3.413 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 5000 into /tmp/models/simple_cnn/model.ckpt.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 5000 into /tmp/models/simple_cnn/model.ckpt.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Loss for final step: 0.13334149.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Loss for final step: 0.13334149.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Accuracy: 0.9021\n",
            "Loss: 0.273015\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "3wGtI-4_LRw1"
      },
      "source": [
        "Our `SimpleCNNGenerator` code achieves **90.21% accuracy**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "EeQG9tuW4RF8"
      },
      "source": [
        "## Generating predictions on our trained model\n",
        "\n",
        "Now that we've got a trained model, we can use it to generate predictions on new input. To keep things simple, here we'll generate predictions on our `estimator` using the first 10 examples from the test set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "dzBtgkgm4RF8",
        "colab": {
          "test": {
            "output": "ignore",
            "timeout": 900
          },
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "0ac2b815-7b60-4acc-83fa-a29c90bce5c9"
      },
      "source": [
        "predictions = estimator.predict(input_fn=input_fn(\"predict\", training=False, batch_size=1))\n",
        "\n",
        "for i, val in enumerate(predictions):\n",
        "    predicted_class = val['class_ids'][0]\n",
        "    prediction_confidence = val['probabilities'][predicted_class] * 100\n",
        "    \n",
        "    # Display the image\n",
        "    plt.imshow(x_test[i])\n",
        "    plt.rcParams['figure.figsize'] = (1,1)\n",
        "    plt.show()\n",
        "    \n",
        "    print('Predicted class: %s, confidence: %s%%' % (predicted_class, round(prediction_confidence, 3)))\n",
        "    print('Actual class: %s \\n\\n' % y_test[i])"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_cnn/model.ckpt-5000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_cnn/model.ckpt-5000\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:`tf.train.start_queue_runners()` was called when no queue runners were defined. You can safely remove the call to this deprecated function.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:`tf.train.start_queue_runners()` was called when no queue runners were defined. You can safely remove the call to this deprecated function.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_cnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b04f95588>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': '/tmp/models/simple_cnn', '_tf_random_seed': 42, '_save_summary_steps': 1000, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f9b04f95588>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_cnn/model.ckpt-5000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from /tmp/models/simple_cnn/model.ckpt-5000\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAQQklEQVR4nO3dW4xd9XXH8d+amTMXxjb24EtdY7AN\nBuFWwrRTkzaoIiJJCS8mUovgIaUSkiMVpCAhtYg+BPWJNk2jPlSRnAbFrVJQqgSBKtRALRoaJUKY\nS4yBhotlGpuxjRlfxte5rT7MBg0we+3h3NP1/UijObPX7H2Wz5yf9znnv/f+m7sLwP9/PZ1uAEB7\nEHYgCcIOJEHYgSQIO5BEXzvvrN8GfFDD7bxLIJXzOqNJv2AL1RoKu5ndLOkfJPVK+id3fyj6/UEN\n63q7qZG7BBB4zneX1up+GW9mvZL+UdKXJG2RdIeZbal3ewBaq5H37NskveXu+919UtKjkrY3py0A\nzdZI2NdJ+tW8nw8Wyz7CzHaY2R4z2zOlCw3cHYBGtPzTeHff6e6j7j5a00Cr7w5AiUbCfkjS+nk/\nX1osA9CFGgn785I2m9lGM+uXdLukJ5rTFoBmq3vozd2nzeweST/W3NDbw+7+atM6A9BUDY2zu/uT\nkp5sUi8AWojDZYEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKw\nA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiC\nsANJNDRls5kdkDQhaUbStLuPNqMpAM3XUNgLn3P3Y03YDoAW4mU8kESjYXdJT5nZC2a2Y6FfMLMd\nZrbHzPZM6UKDdwegXo2+jL/B3Q+Z2WpJT5vZ/7j7s/N/wd13StopSctsxBu8PwB1amjP7u6Hiu9H\nJT0maVszmgLQfHWH3cyGzWzpB7clfVHSvmY1BqC5GnkZv0bSY2b2wXb+1d3/oyldAWi6usPu7vsl\nXdvEXgC0EENvQBKEHUiCsANJEHYgCcIOJNGME2GAjrC++OnrMzNBsbGDOXsuuiisz549G9btut8q\nrflLr9bVUxX27EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBOPs2c2dohzUK/YHs8FYtqTezZtKa0dv\nXBOuu/rfXgvrMydOhvVWqhpHr7L/tmWltY0vNbTpUuzZgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJ\nxtkRqxhHr3L48+Vj6cdHp8J1z6wtP+dbki7765/V1VMz9F2+Pqwf2h7XaxPN7GZx2LMDSRB2IAnC\nDiRB2IEkCDuQBGEHkiDsQBKMsydnfbWw7lOTYX3q878b1k9eXX599tp78X1fuOJ8XH9qQ1g/fGJp\nae2iwfjfdfzgxWG9tuJCWL946bGwfvLdePutULlnN7OHzeyome2bt2zEzJ42szeL7yta2yaARi3m\nZfz3JN38sWX3S9rt7psl7S5+BtDFKsPu7s9KGv/Y4u2SdhW3d0m6tcl9AWiyet+zr3H3seL2YUml\nB0Cb2Q5JOyRpUPH8WABap+FP493dJZV+CuPuO9191N1Haxpo9O4A1KnesB8xs7WSVHw/2ryWALRC\nvWF/QtKdxe07JT3enHYAtErle3Yze0TSjZJWmtlBSV+X9JCkH5jZXZLekXRbK5tEA3p6w3LVOHrv\n8ng8+I0/jrdvwXD0zEA8R/rQkngs2yxev6envF617pVXj4X1/e+uDOvHTw6HdfU1Nj98PSrD7u53\nlJRuanIvAFqIw2WBJAg7kARhB5Ig7EAShB1IglNcFyua2tgrhlEqhr/ksxX1ePvWV/5n9OnpeNsV\n3r5vS1gfqDicqvd8+eN29rK4t4sG4ktNH3wvPtmyp7f8cZ2djfdz42eHwvrsZPw3HVgaDxvW+sv/\n7VXDnfVOVc2eHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSyDPOHo2TS9Vj5VX1SIPTHkfj6FJjY+lH\n//wPwvrk6nise/ne+HLQs0Hrfcvi02vHj8enifrx/rh+Sfn2a33x36TW29jfLDq9VpKWDJWPw09d\nuyne9k9eqq+nutYC8GuHsANJEHYgCcIOJEHYgSQIO5AEYQeSyDPO3sg4uRSek269FZdrno7Hqqt6\na2Qcfey+eBx94sp424OHKqZVHonv34PDGwaH4nH202NL4o0vicfCo8sEnD4Xz040NBD3psrDNip+\nIfDOzYNhfeNP6tsue3YgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSOLXa5y96vrrkaprs1vF/3vBOene\n4PnqVXqv3BjWD9y+trQ2M1RxXvXb8VNgumLm4applydHyh+b/sn4vq1irLpvqOL4hcDMTPz3Pj8Z\nH1+gmbi3C2crzvOfLV//8m0H4/uuU+We3cweNrOjZrZv3rIHzeyQmb1cfN3Sku4ANM1iXsZ/T9LN\nCyz/lrtvLb6ebG5bAJqtMuzu/qyk8Tb0AqCFGvmA7h4z21u8zC+ddMvMdpjZHjPbM6V4/isArVNv\n2L8t6QpJWyWNSfpm2S+6+053H3X30Zrikw8AtE5dYXf3I+4+4+6zkr4jaVtz2wLQbHWF3czmj/V8\nWdK+st8F0B0qx9nN7BFJN0paaWYHJX1d0o1mtlWSSzog6auLujdrcC7xVo5ne/3b7lt/aVg/d/Wa\nsD5+Tfz25txvxGPZPcGp17WJeDx48uJ429NLK861r1VcJ6C//PgGD8aaJeniS+N5yAdq8fNl/GT5\nQQIz0xXXIKjoTRXXhfdzFccv9Javf+x0fHDDqt+/trz4i5+VlirD7u53LLD4u1XrAeguHC4LJEHY\ngSQIO5AEYQeSIOxAEu09xdUbuyxy34bLSmvnrlodrju1JB5qmRyO/9+bHiqvTWwIV608zbRnKq73\nnYmHgTxofXJZvO2ZwbhuVaOhQ/Gpw3au/HGfmowf88n++M5PHFka1mvLyg/PrrqM9ZkTwR9cUm04\nXn/V8tNh/eTZ8u1fs/JIuO7B1ZtLa7O18ucKe3YgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSKKrLiV9\n+k+uj+u/WT5m21MxHnx+ZVz34JRDSbLg0sE90xXrno7HyaeH4/XPr6k4/TbafHCKqST1noifAtEY\nviT1Lokf+J6e8vufqrjc8rkz8am/vafiYycGVtV/TEeVqRPxtMpHZ+MHLhrnX95/Llz33eC4DAue\nSuzZgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiCJto6zz64Y1sQffaa0Pv2n74frn37zktLa4JH4/61a\nfHqxvCceC48u1+y9FZcdrijXKsbhZ2vxv82CofSpiktBV/VWdb575UzYfeXrj6w+Fa57zSVH441f\nGZeX1c6X1vqs4tiF9XH58PllYX31QPyEG5+8qLT27tmLw3WH3j1TWuuZLP+DsGcHkiDsQBKEHUiC\nsANJEHYgCcIOJEHYgSTaOs7eO3FBy/9rf2n9jW2bwvVXb3mvtHb57x2vuy9JOj8dn1t95OyS0tqx\n4/H1y6dP9If1WsV52bMV0yJ7MFbuI1Phuls3/W9YXzUYjxdvGjoW1meCE+IfWPnLcN2/eb/8+uiS\n9NSRa8L6N67699LaSG98rvyMVxyfUOGsx4/7j8+Wz4Hw1vl4iu//Xr6utOZ95Y935Z7dzNab2TNm\n9pqZvWpmXyuWj5jZ02b2ZvF9RdW2AHTOYl7GT0u6z923SPqMpLvNbIuk+yXtdvfNknYXPwPoUpVh\nd/cxd3+xuD0h6XVJ6yRtl7Sr+LVdkm5tVZMAGvep3rOb2QZJ10l6TtIadx8rSoclLfhGw8x2SNoh\nSYM95e97AbTWoj+NN7Mlkn4o6V53/8gZDO7ukhb8RMPdd7r7qLuP9vfEk+UBaJ1Fhd3MapoL+vfd\n/UfF4iNmtraor5VUcYoSgE4yrxhiMDPT3HvycXe/d97yb0h6390fMrP7JY24+19E21pmI3693dSE\ntj+pd0U8GHDqpqvC+vGr4uGvvm3lQ3tXjMTDT5cNx8OC6wbieu/CL5o+NBOcpzo1G79Te+302rD+\n8/0bw/qKZ+JLKq96dG9pbfZM+amazTC7u/w81c+teiNcd+9E+fCWJB0+E5/i+v6Z8lNYJWl6OprK\nOv6bXXV3+fD1z089rpPT7y34hFjMe/bPSvqKpFfM7OVi2QOSHpL0AzO7S9I7km5bxLYAdEhl2N39\npyq/xEFrdtMAmo7DZYEkCDuQBGEHkiDsQBKEHUiicpy9mVo5zg5Aes5365SPLzh6xp4dSIKwA0kQ\ndiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJ\nEHYgCcIOJEHYgSQIO5AEYQeSqAy7ma03s2fM7DUze9XMvlYsf9DMDpnZy8XXLa1vF0C9FjM/+7Sk\n+9z9RTNbKukFM3u6qH3L3f+ude0BaJbFzM8+JmmsuD1hZq9LWtfqxgA016d6z25mGyRdJ+m5YtE9\nZrbXzB42sxUl6+wwsz1mtmdKFxpqFkD9Fh12M1si6YeS7nX3U5K+LekKSVs1t+f/5kLruftOdx91\n99GaBprQMoB6LCrsZlbTXNC/7+4/kiR3P+LuM+4+K+k7kra1rk0AjVrMp/Em6buSXnf3v5+3fO28\nX/uypH3Nbw9Asyzm0/jPSvqKpFfM7OVi2QOS7jCzrZJc0gFJX21JhwCaYjGfxv9U0kLzPT/Z/HYA\ntApH0AFJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Iwd2/f\nnZm9J+mdeYtWSjrWtgY+nW7trVv7kuitXs3s7XJ3X7VQoa1h/8Sdm+1x99GONRDo1t66tS+J3urV\nrt54GQ8kQdiBJDod9p0dvv9It/bWrX1J9FavtvTW0ffsANqn03t2AG1C2IEkOhJ2M7vZzH5pZm+Z\n2f2d6KGMmR0ws1eKaaj3dLiXh83sqJntm7dsxMyeNrM3i+8LzrHXod66YhrvYJrxjj52nZ7+vO3v\n2c2sV9Ibkr4g6aCk5yXd4e6vtbWREmZ2QNKou3f8AAwz+0NJpyX9s7v/drHsbyWNu/tDxX+UK9z9\nL7uktwclne70NN7FbEVr508zLulWSX+mDj52QV+3qQ2PWyf27NskveXu+919UtKjkrZ3oI+u5+7P\nShr/2OLtknYVt3dp7snSdiW9dQV3H3P3F4vbE5I+mGa8o49d0FdbdCLs6yT9at7PB9Vd8727pKfM\n7AUz29HpZhawxt3HituHJa3pZDMLqJzGu50+Ns141zx29Ux/3ig+oPukG9z9dyR9SdLdxcvVruRz\n78G6aex0UdN4t8sC04x/qJOPXb3TnzeqE2E/JGn9vJ8vLZZ1BXc/VHw/Kukxdd9U1Ec+mEG3+H60\nw/18qJum8V5omnF1wWPXyenPOxH25yVtNrONZtYv6XZJT3Sgj08ws+HigxOZ2bCkL6r7pqJ+QtKd\nxe07JT3ewV4+olum8S6bZlwdfuw6Pv25u7f9S9ItmvtE/m1Jf9WJHkr62iTpF8XXq53uTdIjmntZ\nN6W5zzbuknSJpN2S3pT0n5JGuqi3f5H0iqS9mgvW2g71doPmXqLvlfRy8XVLpx+7oK+2PG4cLgsk\nwQd0QBKEHUiCsANJEHYgCcIOJEHYgSQIO5DE/wE8/ft8ncLFKQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Predicted class: 9, confidence: 99.098%\n",
            "Actual class: 9 \n",
            "\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAFkAAABYCAYAAACeV1sKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAALDElEQVR4nO2cf3AdVRXHP+f9TtqkIU1IUxr6i5ZS\noBZESkEZsAqII4ijQCsqDkp1QFAchUGc+o+jzoAj6FAsyA9nmFZGHVQGKmNH5Yfapq3lRwstbaHS\ntE1T+iM/X/Le7vWPczfJvualee+l29d2vzOZfbt77927J9977rnnnLtijCHE0UXkWHfgZEAo5AAQ\nCjkAhEIOAKGQA0Ao5ABQkpBF5CoR2SwiW0XkntHq1IkGKdZOFpEosAX4JLATaAYWGmM2jV73TgzE\nSqh7IbDVGLMdQERWANcCeYWckKRJMaaER/ohsSgATlUKgMiBrvyFqyr16Lh67E6PWj/SdNFneiXf\n/VKEfBrw/qDzncC83EIicitwK0CKSubJghIe6Ue0phaA9stnADDm96vzlnU+cj4AsfZeAMy6jaPW\nj9Vm1bD3SxHyiGCMWQYsA6iW2qJ0U2SMsn/bD+cAcMun/wbAORVvAzAv+RcAdt2vzJ6TSB3Wxj7n\nVQBaHZ2G0kbL3rH5RgDcp04FoHr5f4rp4vD9L6FuC9A06HySvRYiB6VMfDF04luACrcZWGSMyTsO\nq6XWFKIutjxyIQDPX/ULAKbF4wC0Ojrk9zhJADpcZe6EaCcA4yIOCVEVedCq4F3ZKgDikgWgNpK2\ndfR+UnRQ39lyOQD/mzeMfs/BarOKdrN/9HWyMSYrIrcDfwWiwOPDCfhkRtFMLgYjZXLL3RcD8Oad\nDwPwkjUE0q4y2bVaLoLStNqyMm3i/W04KLEco2XHRHp9z3BsG91u0lf3U5UHALjm7eu04IKdR+zv\nkZgcrvgCwFG3LorBY4t/CcC2TA8AGTMOgFQkA8ClOcbDxr4+APpcVbDdbpKm2EEA6qPK9g29NQAk\nxAEGmFtr9XgUHdGvpCsAePiMFQDcMekGALI7i5/TQyYHgLJk8plx1Z/7rWUQt+zzGDx91VcBmLZM\nz59boT9aepSdV1X28m5G6zzbOROASyq2AXDQ6uDLKrTxF7t1JdjmVAMwI7EHgIaoiqZndqP2IWRy\neaMsmXxK1LLLVVs1aq0IjxNn3qUzvtPWBkBSlMETYh0AfHnHFbTOb/e1mdmk+vq2GvUEXH3uxwF4\n5+4z9XjTUgDWWCMkLlp+10e17ckvFv8+IZMDQFkxOZLymw0Za+N6qzNQhvcuVwsg9gl/fc9n0Tq/\nnXcevAiAeIear88u1jZW1CcAqJip59OXW8bfpIeEHTVpo8f4uYdKeicImRwIyorJMn2y/aWeMI/J\nDdGMr9z8uncBaCbqu37Bkm8CMJ5/M/NJ1c+RLjsKrO858vJ/9XTaFADMIb/uzsWC07cA8FYhL5KD\nkMkBoKyYnG4cO+T1qoh2s9NVVl5R/QYAzZEP+8o1rFTLIQvcvOJ5AG6sUl/Ehl41G+5afBsATz6m\nnr2f7LVet6yu/DyrottVO/tjVR6TpxX9XiGTA0BZMbmjKeE7j4jfQ7jL8a/8fmzZduXEuQDIBeqf\n2PHAKTyh5i9PoHr+uk1qU39wlj7jaxerT2LzdzTu8NDCZgBe79PRctBV/l1ZuReAZSGTyxtlxeR0\nvd8l61kXSasnK21Uw9Of7/xK47Ympoz/+sX/BGBl3Wa+t/48AKak9gHwjRr1Pcy64xEAfvao2tET\nz/GPnpT1k3jPHhs5PF5YKEImB4CyYnJPg+s7z9iIsjfjjxHlxOaMetK2f+7XvvJbMurreDVdwbfq\nXvbdeymtlsuFSdW5L2z9l+++Y1d4KTsPZHICRhJTUZlstpBXAkImB4KyYrJb1zfk9UOuRki+uPXz\nADwy/RkAVnaPBwaiHDUR5UxlpJftmWpfG1XW//FKWnM4xkeV9dsy9QBsSavf+L46zeXw7GoPcrYm\n0JjXCl/7lZWQx47r8Z1Pjun5C11qZrWuUHPs9CU69Hdlu33lPed+FAOHBU51UvUCqrUR/Yd2xdQB\ndO+LCwG4b9HbQ/YtPUH/OYnXCnolIFQXgaCsmDxpnLLKm4QaY8rY5s6pAKQO+GejdpvU4rEzQt6o\nPK41yVLWDPSm2BqrRk5tthcW6cFj/l5H1YqJ5G/7SAiZHADKisnTxn4AwAE70dVFVQ+2pHW5vH+W\nnxPdRk25avxpsM4QjI6I67vnHc+yqV85K/j+FIG4LddTr6JKFvRG9tlF1AlRIMqKyUmbvOLmXG/e\nrlaFOzXHYrB61rMqPHZ6LPSVtfe8ZfN+R/X5zLhdsu/2t5205SLiMVmPNYW8kEXI5ABQVkyusGGm\ndE4SZGKrBk7Hz9/ju56bROgxOIo5TC8P6FgdJ13GcwwpYxPbWwFY2a1a9/yklzqr7WRK2IURMjkA\nHJHJItIE/BZoAAywzBjzoIjUAr8DpgDvAdcbYw6U0pn9li5p42ehNQy4oWkdMBCG8hxHwyFuK7u2\nzQzedgYvzVaZ3H3ORABe6lBv/6WptQAccnVl6FQWn2I8EiZnge8aY2YDFwG3ichs4B5glTFmBrDK\nnocYAkdksjFmN7Db/u4QkbfQnU/XApfZYk8B/wDuLqUzPY6yK5VjtLpxPT+/QlMBvDBUSvypAoPh\n6eR4zvVM/8rPX3fHNToq0nvUEbTk1HW2HVuvxqFYFDTxicgU4DxgNdBg/wEAe1B1MlQd3xazkxEj\nFrKIjAX+AHzbGNMuMqA3jTFGJHfN1H9vxFvMeh3tTl3EHxJyZ6i3rcZaE56NO8aysc9qvcH28WBL\nA8DNsTYGmKx1a5o0abxto7o+kx/ytk5YCyaWa72PHCOyLkQkjgr4aWPMH+3lVhFptPcbgb1F9+IE\nx0isCwF+A7xljPn5oFt/Br4C/NQe/1RqZzqzaqNGxc+68TUaOG2wWxMOunq/L4cjXrgqw+E+Cs8L\n5/kwPIZ7IasfzHoBgO9vW+Rr07GDI1pxdHXyJcCXgDdEZIO9di8q3GdE5BZgB3B90b04wTES6+IV\nyOuoHb2N0kBP1r8Z8vSYnicf0j3UrUuVjROiqqO9rbv9sL10kH67uD9BRjyLxPHVnR7T1eTiLZqu\nNeU5q6s194W0HQGxeOEBVA/hii8AlJXvYnxK9aO34vNWdm5Cz5vT6o27uVrn2Kc7NJDqbeUdjFxP\nXNTq4j4zsA0NYE5C22rZp/61M/Z0+ur12vJzT9PkmGKWtCGTA0BZMXnNWt0OVtWkLGxzlKFVr6uH\nbPks9S8sZ+KoPdNrayoahjZzZgHwbkYZXWfV/urXzgBgJmsKfkbI5ABQVkyuX6u6t/ELGqX2klpw\ni19tFQqTUJHURpXC4yJ2E1DnkT1++RAyOQCUFZOr3lf7eEnb2QB80KeMzt08I3H1bZistWmleK6I\nzafoTyTcoBlEn9moK79JY9Wn0bDmKPsuQpSGQD8qIiJtQBewL7CHjj7qOLz/k40x9fkqBCpkABFZ\na4y5INCHjiKK6X+oLgJAKOQAcCyEvOwYPHM0UXD/A9fJJyNCdREAQiEHgMCEfDx+0FpEmkTk7yKy\nSUQ2isid9vqPRKRFRDbYv6uHbScInXy8ftDaRuEbjTHrRaQKWAd8Fo1ndhpj7h9JO0Exuf+D1saY\nPsD7oHVZwxiz2xiz3v7uQL8tclqh7QQl5KE+aF1wZ48lcrKnAG4XkddF5HEROWW4uuHENwLkZk8B\nS4HpwFw0T/CB4eoHJeTj9oPWQ2VPGWNajTGOMcYFHkXVYV4EJeRmYIaITBWRBHAjmoFU1siXPeWl\np1lcB7w5XDuBOO2P4w9a58ueWigic9Gk+PeAxcM1Ei6rA0A48QWAUMgBIBRyAAiFHABCIQeAUMgB\nIBRyAPg/2D6e+eHegc8AAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 72x72 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Predicted class: 2, confidence: 99.99%\n",
            "Actual class: 2 \n",
            "\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAFkAAABYCAYAAACeV1sKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAIT0lEQVR4nO2ca4xUZxnHf8/Mzuywyy4Ly6XLskDT\nUppNSkokBSFtNFrT8AWrSdOqtR+aVA1N2sQYiDFR4xc/qLGJ0QRjk3qp1apJ0dBWpdQICgEaulwq\nLUU2wgLCctvb7NwePzzvWZjZnWXZmfPigfNLNnNu7znPPPuf57zv815EVYkJl8TNNuB2IHayB2In\neyB2sgdiJ3sgdrIHanKyiDwiIkdF5JiIbK6XUbcaMt16sogkgfeBh4GTwF7gCVU9Uj/zbg0aaij7\nAHBMVY8DiMgrwAagqpPT0qgZmmt4ZDml2XYvmZcHIDeSshMNJSRnP1INfqtJJyb3kU4XrOwHuZrt\nyDJETkel2vlanNwJ/Oea/ZPA6sqLROQZ4BmADE2slk9M/4nivof79Q190h7X+OXTAJw4tBCAxPws\niX/PAKDQbNdqm/0jNG9eX7LknJX91Inp2+PYo9snPV+Lk6eEqm4BtgC0ypza2vDiZKlFAFZseheA\nH3futuPdVy/9cN0gAB3JNABNCfs8XXDHG2YCsPrJrwDQ9ot/1mTaZNTy4jsFdF2zv8gdi6mgFiXv\nBZaJyJ2Ycx8HPlcXq6pRKpbtbl7wVwB6cvY19o4sBaAr1U8mYUrdPzoLgOFSIwAJ5gLwxdbzAFxa\nbvdqC8/q6TtZVQsi8izwJpAEXlTVw3Wz7BaippisqtuAbXWy5YZZ7OLquVGrISxrPANAmiL9Jat5\nZMReeO0pi8X9xZll98h11l67uB5xi88DodcuwqBh6WK3dQCAgVIGgCJWxUtLcUzBQ2qxOK/2VUuu\n4vxh3pQ9Z+5A6PbGSvZAJJV8eVVH2f4Vp+Q7Gi4DkNUUWbXWX1qsRpKgBEAmYQrvd7WNu2b32z1D\ntDdWsgciqeTzK0wbl0sjAJwr3AFAZ8MlANoTIyxrsJj7bq4dgJLTU6Do9sSolR2x2kaa/tDsjZXs\ngUgquXmlqS6vpsrO1EUAhtTyE8tTWb559iEAvjF/JwAH800AZF09Ochp9PaZ0pfRG5q9sZI9EEkl\nf3aJZd8GSpbUy2kSgG4Xh98amc+hj5jKZ/eZgtN5q2WkxPLIQVZOLqZDtzdWsgciqeTlGUvSDzsF\nB625IJexft+jdFKeq8o4BWdLgXKzAJTSpbDNjZXsg0gqeW2mD4C+oqkyyFkEtLzaMrZ9sTgMwH1p\naxXuzza5M1fsY0Z5jjoMYiV7IJJKDvrnegumwmbXegtoe62HINI+d/IRAF5Y9AZwNXcRkLyQCtFS\nI1ayByKp5EpaElZTGC5ZL0dpeHjs3L5Tlntu7LKvmqS8NpG6Er7OYiV7INJKDmoVrWIx+ZcDd467\nJttnfX0pSboy/nUVaScPucR7V9rCw0u9awCYyfGxaxa/buFh+DMWSoJmtU/icOGBSCv5ateS0ddr\nact7rlFy066jAMxK2Ni4VveSDGgYJnRiJXsgkkp+Y9hi8ULXcZp3wxgbz4xvWGiufPBKMFQgoGEo\nBAMriJXsgUgqeefgPQB8vm0PABmXHyrcPTLu2lK2PAYHQwXAqn2FJkInVrIHIqnkVw6vAmDjgzZw\n+0LJGhrrlx8C4OgkZeckB92WKTo5Wv3aehEr2QPXVbKIdAE/BxZg01q2qOoLIjIH+A2wFDgBPKaq\nF8Mz9Sotu6zOm3nINDLgupS+veBvADzO2nFlRtVqFRkJkvSmZAm/92lKSi4AX1XVbmANsFFEuoHN\nwHZVXQZsd/sxE3BdJavqaeC02x4QkfewmU8bgI+5y14C3gY2hWJlBR1v21SEc5vcLCg3qOUfo9Wn\nrx3Pm5KTFV1V6iFg3tCLT0SWAiuBPcAC9w8AOIOFk4nKlE0xux2ZspNFZCbwe+B5Vb0iclURqqoi\nMuH0sbpOMXMUj7wPwAd5y1W0J6zZNi9pn4kV91Lq+VdZmQFXP26uyMK5UQWhMqUfi4ikMAf/SlX/\n4A6fFZEOd74D+G84Jkaf6zpZTLI/A95T1R9cc2or8JTbfgp4rf7mTU57Yoj2xBAZKZCRAnMS9ndl\n+axx1+4Y7GbHYDcZKZKRIj25LD25LJoMX81TCRfrgCeBgyJywB37OvBd4Lci8jTQCzwWjonRZyq1\ni51AtcnZNUyUngYVc6u/sPtpAP6y7keATSYEOLNWuPvV8qKnRsunQybdTPbGi+GvJha3+DwQrdxF\nxQT2eX+yoVfND5rCg6G0Gx/+M2/SWlZ0RtLqyUHna/CZHI2VfEsQKSVL0qKuuonsrS/bEgwHv2Oq\nbU9Yh11+gurC1mP3AfC1NbsAOFt0rcUO09n4+kj9iJXsgUgpWQv5CY//8dJKAH7YsQ+ARQ0HeH39\n8wA0btsLQDJp6ba5SctvtLiBh6PtcUy+JYiUkqmy8tdbLz8AQPdH7wWg7Xczadm2u+yaWb+24bYf\nb9kAwIUhS1Yt/Hv4I4piJXtg2uvCTethIueAIeC8t4fWn7mMt3+Jqs6rVsCrkwFEZJ+qrvL60Doy\nHfvjcOGB2MkeuBlO3nITnllPbth+7zH5diQOFx6InewBb06O4oLWItIlIjtE5IiIHBaR59zxb4nI\nKRE54P7WT3ofHzE5qgtau174DlV9R0RagP3Ap7H+zEFV/d5U7uNLyWMLWqtqDggWtP6/RlVPq+o7\nbnsACEZP3RC+nDzRgtY3bOzNpGL0FMCzItIjIi+KyOzJysYvvilQOXoK+AlwF3A/Nk7w+5OV9+Xk\nyC5oPdHoKVU9q6pFVS0BP8XCYVV8OXlsQWsRSWMLWm/19OxpU230VDA8zfEocGiy+3hJ2kd4Qetq\no6eeEJH7sUHxJ4AvTXaTuFntgfjF54HYyR6IneyB2MkeiJ3sgdjJHoid7IH/AZ6vj8FacP9xAAAA\nAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 72x72 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Predicted class: 1, confidence: 100.0%\n",
            "Actual class: 1 \n",
            "\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAFkAAABYCAYAAACeV1sKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAIS0lEQVR4nO2cXWwcVxXHf2e/s47jxE3imNSto9Rt\nCAVSVAgQgSoBAqGK0j6EJlJVCVArRBEVCLXiiRckHgoSvCAFNRKRQHxWah+CikhSpKZfbqw0adKQ\nhDRpnDqO3SZ27M2ud3cOD+eOP2PH9u7eMMn8JGt2Zu/MnDn+75lzz70zoqrENJbE9TbgZiB2sgdi\nJ3sgdrIHYid7IHayB2pysoh8TUT+IyInReTpehl1oyGLzZNFJAkcB74C9ALdwDZVPVo/824MUjXs\n+xngpKqeAhCRPwEPALM6OSNZzdFUwykNyaQBKK7O2HrVbQ8mtalO32nqMlG2ZfJSAYBaOmVFRhnT\nksz2fS1OXgucnbTeC2ye3khEHgMeA8iRZ7N8qYZTGqk1twJw/AcdAKSH7fqSxYk22SHntHCRtGU1\nY23zF+w/0vL8QQCC4qSdF8jrumduexd95HmiqjuAHQDLpLUuffjTj9wGwBvbngHgrbGlAOwb+SgA\nD7d08+LoRgCe670HgO0dbwAwWGkG4NlXvwBAoe1TALT95pV6mHZVarnxnQM6Jq3f6rbFTKMWJXcD\nXSKyDnPuw8D2ulh1DYor7af+15E7ACgF6Snf7y3cRdoF5a6WAVtmzwPw3+JqAJavuQxA5URrw+1d\ntJNVtSIiTwAvAklgp6oeqZtlNxA1xWRV3Q3srpMt8z9vq6UGBy7fDsDWWyzeHi5a9OrKnOfUmCn2\nzqZ+AJLuDtiZGwQg0A0ArDo01nB74x6fBxqeXTSC9FnLjysbLC8rq11GGJvPlm9hqJoHIOsS4n+P\nmHI/mX8PgISYsnM97wIwPa2uJ7GSPRBJJTsRsu/4nQD0Fy33DdX5UFsPdy+xflLaaXTX0JYpxxg6\n0wJA+9j7Dbc3VrIHoqnkwLrGqXNZAN49ts6+cAr/5/0l7m42hT7ZehiApwbWALD/cBcA+T7Xz16S\ns+XwcMPsjZXsgUgqecUx6/H13WfL5cdMlamSSfm1k+s4esDqGN/9SQ8Al95fBsCSc3bJ2YvWVi+P\nNNzeWMkeiKSSy3mLyYmCaaSwxtazFyfaBO7KViRczHXV3mTJlomK29zk6tuFQsPsjZXsgUgqOT1q\n8TTIW0yWwCm63Sk8pazusSJ8gGuzxKQbpO2SwyI+pVLD7Y2V7IFoKrngBvPcIj0SDriYkosjKZIv\ndU/dJ2dKDoeowtgcxEq+MYikkqVqypWyC6yhkN0yMZocbztQNaVmMqbkMJ4nKi5PLlcabG2sZC9E\nUsm5fstpRa36pgmLxeFQX3LSFIhTFRvJFlehC2Py0l43IhI0spJsxEr2QCSVnDjdB0CQcb01Ma1U\n8i5/nnRVTWKKLRSsYtcybClJqH4fRNLJwZCVJZOuWz2ewrnfZXVFebzt2YoN+Yc3vmTRYkq23wpD\njQ8WcbjwQiSVrBVTZapgP3l1Uglc5pYanJjs8sd+m57XlLOwEWRsgDXIZ3yYCsRK9kIklRwS3uBK\nrU7RKbupZS5NaKf7RCcAa9utDlpa5gr8o6bkqRO8GkOsZA9EWsnlVssNWk6aOpu+bJMLEztXjrdZ\n9palbvd+wia1vH12OeA3hYuV7IFIK/mhT78JwGudnQD8beMuAL69+6thFZSVh6xA9PF8LwD/+J5N\nDk8cse72bXsbb2esZA9cU8ki0gHsAtqwYuIOVf21iLQCfwY6gdPAVlW9ONtx6oq4gVM3GvqNtTaB\nZdeQPboQjI6ON02UTdMbsjbZ5fGPvQzAzvTnvJgK81NyBfixqm4EPgt8X0Q2Ak8De1S1C9jj1mOu\nwjWVrKp9QJ/7fFlE3sGefHoAuM81+z3wEvBUQ6ycaRQALakrALSlhwD40JU1J1PNmo6axOoZd+VM\n0emkj6qFsaAbn4h0AvcArwNt7h8AcB4LJ1fbZ8ojZjcj83ayiCwF/g48qarDIhN5pqqqhFXxaTTi\nEbOQocoSAO7I2iMLvcHMh2ySLiYX3UTx5oRV7S9dsIL/6noaNAvzyi5EJI05+A+q+pzb3C8i7e77\nduBCY0yMPtd0sphknwXeUdVfTfrqBeBR9/lR4Pn6mzc3A2PNDIw1syo1zKrUMIPlpQyWp8blRKlq\nfxKQkIAMVTJUoSL254H5hIstwCPAYRE56Lb9FPgF8BcR+Q5wBtjaGBOjz3yyi5eZePx7OrU/KF0D\nW1pOAJBzmUN6xlPrkByyDKTs5mXlxHLrxBV//bC4x+eBSNcuXh1eD8Dn2+wxsSvVcLRj0tSrCx8A\nUFSrHLe67CI3ECv5hiLSSt67bxMAP9/+LwDKOlMz1Q8+BOBEyR7MWZW3ukbze8GMto0iVrIHIq3k\ncPrryqRNchkLZr+cNa6+0ZE0BTefWfybWhZKrGQPRFrJ63fYuN0XNz8IwMU97QB8hJmvuvnR/m9N\nWe/a39Ng6yaIleyBRb8XblEnExkARoFBbyetPyuZaf/tqrpqth28OhlARN5U1Xu9nrSOLMb+OFx4\nIHayB66Hk3dch3PWkwXb7z0m34zE4cIDsZM94M3JUXyhtYh0iMg+ETkqIkdE5Idu+89E5JyIHHR/\nX5/zOD5iclRfaO1G4dtVtUdEmoEDwDex8cwRVX1mPsfxpeTxF1qr6hgQvtD6/xpV7VPVHvf5MhDO\nnloQvpx8tRdaL9jY68m02VMAT4jIIRHZKSIr5to3vvHNg+mzp4DfAuuBTdg8wV/Otb8vJ0f2hdZX\nmz2lqv2qWlXVAPgdFg5nxZeTx19oLSIZ7IXWL3g696KZbfZUOD3N8SDw9lzH8VK0j/ALrWebPbVN\nRDZhk+JPA4/PdZC4W+2B+MbngdjJHoid7IHYyR6IneyB2MkeiJ3sgf8BM3eYOF9QaUgAAAAASUVO\nRK5CYII=\n",
            "text/plain": [
              "<Figure size 72x72 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Predicted class: 1, confidence: 100.0%\n",
            "Actual class: 1 \n",
            "\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAFkAAABYCAYAAACeV1sKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAALrElEQVR4nO2ca4hd1RXHf+u+5857MpnJxJlkJEZN\ngmkUG2u1VWqlraXYQpVaKIUW7IdKK0hR/OQXoR/aQvulJUWhhUBbUNAPLQqhiuKjPmo1GmNinGhi\nMnnMTOZxX3Pv3f2w1r537nVed2Y8uUnOH4Y955y9z9ln3f9ee6211z7inCPE54vI+e7ApYBQyAEg\nFHIACIUcAEIhB4BQyAFgVUIWkW+KyEEROSwiD61Vpy42yErtZBGJAh8AtwPHgNeAe5xz761d9y4O\nxFbRdjdw2Dl3BEBE/gbcCSwo5IQkXYrWVTxyfriOtJZR0VIAqatjx5GikioykVmz5+eYoeDystD1\n1Qj5MuCTOcfHgBvqK4nIvcC9ACnS3CC3ffZOkaiW5ZJvVHt9idGWv/mLABRbVfuVEkI5WnuPUlLL\n9Gl9RstT/1n0no3gVbdv0eurEfKy4JzbA+wB6JCe+aXlylouIdzouh4AznznKgDO7tLr37/lFQD+\ndXSbNRO601kAtnaeBuDgRB8AHSk9n3hgPQDvP7sVgOEntF7pwKGG3m85WM3EdxwYmnM8aOdC1GE1\nE18MnfhuQ4X7GvBD59y7C7XpkB43r7qo3lTLuj4df/DLAMxsmdUTUb3eeigBwGyrHievmQAgl4vT\nls4DMD2T0jpTWleyqpoivXq9NK2DeePmswBMZrX+0K+U8aXDHy3cX8Orbh+TbmztdbJzrigi9wHP\nAFHg8cUEfCljxUxeCRZk8gIMPvawMjjfrecT57RepGjVTdnFp7X0k9vQHSMcOb0OgEIurnWNwT1v\nKK8yt2sjd6ANgGKLto0MzQAQtdGy6a53lnyvpZgcenwB4HO3LpYFsd/aqXkVveoKADKDStm2Ee3m\nbJ2JHTNTN9unrOs4osfvf7yBncM6Bx+d6AYg96FaJuM35bTSSbWto8b+cotaOGVjfk/fOQBGf6Gj\nqf8PL1lf5x91iyFkcgBoDiZ7J8QwtUP1qdjpsvkqcVWXlNRQwNn52IyyK9+lx33PJrj+waMATORV\n2WZi5ulZmRjVxkWzTGhXyyUaV0ZnCsroqZ0FAPp951Ywh4VMDgDNweQ6nN2uLItmlTVl05soqYip\nCVuxLiLG+Nl2LeUEPPbSV/X/orI8WTJdeiRllaxNnzHYGJ5Mzdb05Vtf2A/Ah6t4n5DJAaApmZy9\nTK2K2KQyupT0elDp13JKj4tpPS7bW0SMhGPbha79ejKvxgXpE9omM6BtCp2qe7vXTwEwPtoBwI1X\nfADAy8eHATh4TmMeidQpfVYu1/D7hEwOAE3F5NjwpprjUquyTbpUGcffVUvBM9ZbFx5iwbxoXiqW\nRilho8B0cDmux26Dxi5yZkVIUhX7la0nAXiZYe2T3bRw0w493vdGw+8VMjkANBWTZ7apNRqdVoqW\nU8qilrSZFU6ZnO8xXawkrMQsPFykamOLETm73ntqWsQTqvfjsZI9S4fH0WyvPjOhx/mSimhqixrn\nvYvH5+dFyOQA0FRMntqk3Ynm7USkNrCVGVTWpT8xO9oI7j3CSlQuU9XbM4P+mnl8ZjcXCvqsWExH\nyxV9ZwA4ldeoXL4YtVJHT1bDKfSu4L1CJgeApmKyt2GlrKyLTykHvH7MWHxBjLoV+9j0b8koI+Wq\nnnYWF45P68VCl3mROa/39Zmj0+oubulWRk9PKoPFlHrfNadW/F4hkwNAUzHZr9V5fZrUJTtu3KDr\nbC/805b+26yBWQq+vo/OlZJVlvvYhdfbFe+xbLo5ryZK8ZAyuffrR2uul214dCZzcx/ZEEImB4Cm\nYrK3i6M5/e3F1vIiphd739Hw2/FbVF/6+LKH9/gKnY7EhOl3z3bT25FZY2jdilynpVts+PY5ahoW\ndAhsbhsDYKTx12oOIUtcx7nzLq93l00QM0WdxRIjmoDCrbXut18E9Q5IMe1IjtU6H770P5wXYiRi\ngaKD+gMOxFVHScT3RcuNSRX+J93qMJXGx5f9fqG6CABNweTo4EDNsR+pntFTxmTi2t3qpKZl2Sa8\nhJErlpVKnVLKT6amPvxKlw8YlZVn8ZFRAHIuXtuXvF4vWwO32foaMrm50BRMLvVqwJyYn/i0W4VO\nZeHbJzYCMDyuy/zFtLJJvJkV87Nbla0lW2XyS/1ipph3q8UWTFtbzTTrVLvw9clhPbZ7V5wZGwLZ\njZqXkHxr+e8XMjkANAWT/TK7zGh3Ihb4yW3VSJEzR8HP6JFSXXOf3uzVqasGmSJZy1lO+bquUgfg\nsk61GmRSmfv8QU2lbWnTG2QnLKnRHlJo17IuurooQiYHgKZgcq7PlpXMWvCsa+tQ29Xt1xOxIY1b\nFtvM6zB2eSZ7tzqWkWrQvlx7Tx8OdbPGcLMuSlt1wTT5kVZs3z0JQCal6VxPj1wDQLRjwbzCBREy\nOQAsyWQRGQL+imYqOWCPc+73ItID/B0YRr3Nu51zyzce5+DUdapMSy1FK5V+O9aphzcyrquiY19R\nJkcztSFR6uzqyGz1/2qplaJ+/0xB+eWXl8auVQbHNUOA6ZxqXWnRIdFl2yNGrlNLaF0D77ccJheB\nB5xz24EvAT8Xke3AQ8A+59xWYJ8dh5gHSzLZOXcCOGH/T4nIAXTn053ArVbtL8BzwIMr6oTf7ZVU\nBvcPaPxgMK3l9OuqH89cryyKWzJ4MV3ppRbei0tUGex1s09K9EtWOUvbKppOntmkzx5+WuOm+x7Y\nC8Du/94FwERG543UycansYZaiMgwcC3wKtBvPwDASeYkPta1qdlidili2dsZRKQNeB541Dn3pIhM\nOOe65lwfd851L3aPJTfmGKLdepvZHZsBiLykWwo+enQ3QCWM6WMWs+36DukTej63zlXTBSxBpuXT\n2kyYbL+ej6xXj2/noHqTuZ+o5yc5pXx5XEdTeaYurjoHa7KdQUTiwBPAXufck3Z6VEQG7PoAsPJF\nsIscy7EuBHgMOOCc+92cS08DPwZ+beVTa9Wpimf3opZ+k+RstyrY5LhF4yz+nDpt+tXiyoWeUkU/\nS8Ebxlp49nsrw4/j9SndqPO/G7YA0Ln3lbV6nWXp5JuAHwHviIgPizyMCvcfIvJT4Chw95r16iLD\ncqyLF/nMdvAKllawjcA2vUjUPLmi2s3j37hSzxfrtgFnjY1G1szlahm0HK3GhHP93vXTwseXfdt8\nVkXw5indXHvmZtXVnXtr+7SSbQweoccXAJoidlGBscUz2GPiSuVCbFKvl4yoPi7htwO3fKwXYhnI\nbrB7JbSSE0soT9Xa1FgMIzerokj31VkRnsGrYHTI5ADQXEw2SEy75RmdG1KbNZKwzZQfWgqVxYzb\nDiuDk2PKsomrXWXVBLNAPOtj03o+MenTb2tTwbrtUw2RtDpO5Yy5o3UbOhtByOQA0JRMduVavXf1\n/QcBOPSIbinY9jXNRNnSpsmBz32qqxkFS3ftT+UZPdsJQG+n6tipNo2qDXRpmG1nj3p4IzMaTxsZ\nt+3Bf9L1xFTmWG2nyo0z2CNkcgBojk8xrBDRbcrg8V3Kxky/cma2vWo7e/ikRL/60v6xKumuZ3SU\nNJIRVI/wUwxNgECZLCKngRngTGAPXXv08tn+b3bOrV+oQaBCBhCR151z1wf60DXESvofqosAEAo5\nAJwPIe85D89cSzTc/8B18qWIUF0EgFDIASAwIV+IH7QWkSER+beIvCci74rIL+38IyJyXETesr87\nFr1PEDr5Qv2gta3CDzjn3hSRduAN4Lvoeua0c+43y7lPUEyufNDaOVcA/AetmxrOuRPOuTft/ynA\nZ081hKCEPN8HrRvu7PlEXfYUwH0i8raIPC4iiyb1hBPfMmDZU08A9zvnJoE/AluAXWie4G8Xax+U\nkC/YD1rPlz3lnBt1zpWcc2Xgz6g6XBBBCfk1YKuIXC4iCeAHaAZSU2Oh7Cmfnmb4HrB/sfsEsvx0\nAX/QeqHsqXtEZBea5TUC/Gyxm4RudQAIJ74AEAo5AIRCDgChkANAKOQAEAo5AIRCDgD/B+aNHoBf\nU3GoAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 72x72 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Predicted class: 6, confidence: 92.3%\n",
            "Actual class: 6 \n",
            "\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAFkAAABYCAYAAACeV1sKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAJQElEQVR4nO2ca2wU1xXHf2cf3vX6tRg/sAHHCSUQ\nGjWkikIjgkSgtClKlKaV0hC16oeoaaWiUqkfSF9qv1SqqjZVPlUlatJErZpEKVVpQ4oUIJSmKeUN\n4RkEBMzDmIeN7bW9r9MPZxZj4zXG3hk6MH/JGs/svTNnz/z33HPOPfeKqhLAXYRutgC3AwIle4BA\nyR4gULIHCJTsAQIle4AJKVlEHhWRQyJyRESeL5VQtxpkvH6yiISBw8ASoA3YCixT1f2lE+/WQGQC\nfR8EjqjqUQAReR14Aiiq5DKJaZyKCTxy2P1m2w8xjwCQy9t5OJQnlS4DIBQyEsUjGQAy+TAAgjpH\ngx7OjFuOfnpJ64AU+3wiSp4KnLzqvA2YN7yRiDwHPAcQJ8E8WTyBRw5F86tVAPTlogBcTscBSJb1\nsePUNADKY6a8WXXnAGhPWZ+yUA6wFwKQe+T0uOXYoutH/XwiSh4TVHUVsAqgWmpLEsOH6yYDsLzx\nHQD+nZo55PPm6CWebtgCQEe2GoB+tRdxpiIJQG2kF4BtXXcAcKEUghXBRAa+U8D0q86nOdcCDMNE\nmLwVmCkid2LKfRp4piRSXQdSkQAg51jURGgAgK6cXe/MJdifagYgFsoCMCNu5iKv1mdd+xzrM2Am\npoZLrsk7biWralZElgPrgDDwsqruK5lktxAmZJNVdS2wtkSyjBnHnzErdZ85ELx92Wx0IpQGICpZ\nOjPG6ogzwDWUGWMTYWvTUmnMbZx8GYAPFj1o7TdsL7m8QcTnAVz3LtzAk1/ZDMCmPmPr7k5z1+5P\nmkeZysdYMsks19lsDQADefMuzmcqgUG37+HKwwD8Zd4CAKZtKL28AZM9gC+Z/K3aDwD48ekvADA5\nZj5vTSQFQFRynMzU2rVwHzBor4+kGgBo6zF/OV1vEWBfU841eQMmewBfMTlyVysAcXkfgAMXGwG4\np7YdgIwaK7tyCR6r3AtAR97s9vF0HQAVEfOpY2HznwsRYWjAPb4FTPYAvmLyQIvZ2bbsULFDTkbt\nXNpYObfiBD9pexyA5c2WvGmJXgTgWMRsciExlMrH7HzAPbkDJnsAXzH54mxjXa+TUbucsigOI/iV\nvMSi8o957WGLCv+5ZzYAzya3AfC3rPXpy9o9+tVUEC6eDp4wAiZ7AF8xuetus70nM5arqE70A4PR\n27yaswBsHWi40uf3ex8C4PsLbcKm4IFUlZkRzqvxLDT+iZHrImCyB/AVkyvu6gLgUH8TAOVRo19/\nzr7GkoTlIRZvXMFMLJvW8ooz7/eIHQv55QJSeUvliXsBX8BkL+ArJtdXWo6iI22Toep4E3EneqsK\n2fmsF3rJO32i7xqjM2pUjTqUTeec6DBbDgRM9j18xeR+J9I7229MLvjFDfFuADb1ma3O7zl4Td+d\naeN2SMxDOdVleebZNZb3yMXdkjpgsifwFZM7LhmD45GhHkJLzPISK7d+GYAZ7Lym76Zei/wKfnLP\neatkOlhjmTx1kW6+UnKmx9ytVNKCj1jYRquv1lha8601nxtsHDJlkrc2/zj7SQAeqjsGQOSCffVD\nkSnWburQF1dKBObCA/iKyWRsoKt2QuLGhE3nR50il+TODgBygETtq+mAMfnYIRsUH51iE6zRbuuT\nrbNjtDPsmtgBkz2Ar5ic3GfiTr7PgpJk1CZJX+m6F4D8sauKTHNDo4uWtebCLXt8NwAvVZj9Ttb3\nANBzcZJLUgdM9gS+YnLjb/8LQHZZoWDFxP9EzFKcb33J2Fn1xn9AhvKnYrfVH/+9ZxYA4sTdIWca\nKlsdlAT4Gr5ismbNl01lzV9uLrfUZ2EytGeZnVe9AZpJD+mbbbPS6QWJIwD8Yrp5KHUJK4jp7K91\nTe6AyR7gukwWkenAa0AjoMAqVX1RRGqBN4BW4DjwlKq6V0l9FZrKzT9uiFpiqFCgsmLWRgDeZErR\nvvVhs8FL55i/XB0xD+VwvNkdYRkbk7PA91R1DvAZ4NsiMgd4HlivqjOB9c55gBFwXSar6hngjPN/\nt4gcwFY+PQEsdJq9CrwHrHRFymF4d7vlIV5c8gcAdqZaATiRK9jV4ut/VnffDcC9FW0AJMNmk/8U\numbhVslwQwOfiLQC9wNbgEbnBQCcxczJSH2GLDG7HTFmJYtIJfBn4LuqellksBhEVVVERqSPG0vM\n7vn1eQA6F9lLK6QvZ5fbO//wUwtHTNwDHBuoB+DOmOU54k4tQKTTPUdrTN6FiEQxBf9RVVc7l9tF\npMn5vAk4546I/sdYvAsBfgccUNUXrvpoDfB14OfO8a+uSDgCch8dBeBgn3kEU2Pm1BTsa/v8SdTv\nGblvt1OmlSg3PzkZsj65mHt7MY3lNzIf+BqwV0R2Odd+gCn3TRF5FvgYeModEf2PsXgX/2Jwnfdw\nlG6h9FhQGAecnQ1ef99KsH642H5Enc5iSVl6AX4z8i1OpyzvUVZdKBFwZkRC7jE5iPg8gK9yFxI2\nL6KQw2h5x6K38Gft2J4xlj7QeJLjRe5xuseiw9qw5ZF39dsCdpmULtJj4giY7AF8xWQdNtsRe3sr\nABt+ZNP9MxLmP8+v/oijC2w5Q2jz0PKAzm4ry5oSsbxHd97OtbPMJakDJnsCXzGZIvsl7ThjSxdW\nfnodAL0a4cTnzR9u3Ty0bU2lFY5PcWo2KLMYKlrfV2ppryBgsgfwF5OLYNrP7PjYN1YAIBmh9b0i\n3sJqWwoxr+M7AIS6rBpp6sb8yO1LgIDJHmDc+8KN62EiHUAvcN6zh5YedVwr/x2qWl+sg6dKBhCR\nbar6gKcPLSHGI39gLjxAoGQPcDOUvOomPLOUuGH5PbfJtyMCc+EBAiV7AM+U7McNrUVkuohsFJH9\nIrJPRFY4138qIqdEZJfzt3TU+3hhk/26obUzC9+kqjtEpArYDnwRm8/sUdVfjuU+XjH5yobWqpoG\nChta/19DVc+o6g7n/26gUD11Q/BKySNtaH3Dwt5MDKueAlguIntE5GURGXUtRDDwjQHDq6ewufAZ\nwFysTvBXo/X3Ssm+3dB6pOopVW1X1Zyq5oGXMHNYFF4p+cqG1iJShm1ovcajZ48bxaqnCuVpDp4E\nPhztPp4k7X28oXWx6qllIjIXq9E9DnxztJsEYbUHCAY+DxAo2QMESvYAgZI9QKBkDxAo2QMESvYA\n/wN8vuIlzvHMeQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 72x72 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Predicted class: 1, confidence: 99.995%\n",
            "Actual class: 1 \n",
            "\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAFkAAABYCAYAAACeV1sKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAKT0lEQVR4nO2cbYxcZRXHf+femd3Zt+5ud5d2S1tK\nW0CgBogoqEQxYmIwWvxSxWj8YFJRayRRI/GTH/0gRmOisUQSMQZpgomQEEyoQIAgAg2CvAilFtva\nbbtt6e7s7uzOnTl+OOdOO8NOme5cbqfl/pPN3fv2PM8987/n/s95zr2iqmR4bxGc7QG8H5AZOQVk\nRk4BmZFTQGbkFJAZOQW0ZWQR+ayI/FtEdovIHUkN6nyDLFUni0gIvA58BtgPPAvcqqqvJDe88wO5\nNs79CLBbVfcAiMifgM1AUyN3SbcW6Gujy8UhebsMLUe23lOgmveb1DkkMZmKc4n3X2KGBZ2XZvvb\nMfKFwL5T1vcD1zUeJCJbga0ABXq5Tj699B7Fr6Ph7suNrgAgmjgEQHDZ5ZRW2o8ZRHZsWKpYE0+9\nsPT+m+AZ3Xna/e0YuSWo6nZgO8AyWb4k3yQ5Z2oU+QYz9v37ngagPygAcDAqAnBBuIvXyvMADAZm\n3OmqMfvmh28H4NLb/rGUoSwJ7Tz4DgBrTllf7dsyNKAdJj8LXCIiF2PG/TLwlURG1YAagx1Hbrse\ngKvvteXolUcAeOqqHQBsfPA2PrzpTQB2rLdbecsec1OX/+oEANV8l7VdXrBGm7iiJLBkI6tqJCLb\ngL8CIXC3qr6c2MjOI7Tlk1X1IeChhMbSMgIndu+EsW/wh7sB2PzY5wD49g07+dKyf/rR/QBMfXcl\nAPpyEx68hynfLOJLAe+5ujgjNPjFoGCqoXTjB205HAJQ6bbjihebctCPXgVA+UZj7+a3XuRvs+sB\nuPcDq7xxY7Bcu8naKNild+01f05XHoBor6vSaiWxy8qYnAKWHFYvBctkuZ5JMFL95DUAHPhEDwBD\nr1cBKBw3p5wrlgEorrX93W8b+6KegIEn9wAg3a4i+uyY+C6Z3bjc120Rznvbr09YG/tbV6PP6E6m\n9FjTiC9jcgroLJ/cgNKIsXDwTWPZQr+RpfeQrw/Z/p4jxujuCYv4Dn9sOdVPbQBg6Dlj5sKqZbYc\ntEvunjR9nD80BYBUrM0a4xNExuQU0JFMDoeHAZgfMA70HzDWHfqCDXd0l+Ulop7euvPkhDG50jVC\nOSbkgrG8GsbKxRa5orUp0zO2ecASStWBQpKXAmRMTgUdyWTG7Mlf6bbVwj7LN+TGjG2lcWNwftrz\nx5H705lZALqmlLAcJ5KNwd2HjbE9R80HEzi/QtPe6iqkmrf1sNf6qM7Otn05GZNTQEcyORqxfEPo\nCTImjwFQPuEMLzg7Jz2JEbi/rVR8P+TnjMnqTJRe87Wxeoh9cbRmzLY7g3OT5tdlxPrKmHyOoDOZ\n3GfDChfcr8ZzdwVj6uyo6+ODxpF4/u7U6DWedsJnVQj92DlTJlo21RHM2lLK9XN/mgCDa2NJrKUM\nTdGRTK70mH8UT4RViuY/R8dMGcwOmx/NHTX/WV5h0VzoWTuJ7A+AuZI34grEGV2ZPArA2zdfCsDI\n3w/bcVqvSpJAxuQU0JlM9nxxfsbYJ64eVvZPAzAxMQpANGoqpMY+zwGHZUXDeJfnpj3y03z9Jc/c\nYnfH8CumOoL//M/67K2PJttBxuQU0JFMjpzJhaPGPsS48NVxq7P4RWkdAMGcOd5Kn81q1JQEoGG9\ndo59MSem6/p68EO/BWBrYZt15T5cBpclcSk2rMRaShDid3/+uF+wBxBb+i28/vWUGa7aY8Ov9Noy\nP2/yDAWpuqyL5Z+3rSc8rPYH2+NzNk1VXGsPzWVPevASJneTZ+4iBXQUk4M+SwCp//Qy5+nI5cN1\nx0V9dkB+2tgYlF2elYzJ1fxJ+SU+QRozu1KyuyO8xBh8U+8TANy52tqsOYkgY/I5hY5isjTIK46Z\nD567am3d5tysMTcsmb+NchZmB8ND1k4FNFj8wSdenlV5wyZaH5+7CIDi+vpSsFgWvqPYcQnImJwC\nOorJNcSxxZQpgcPXWva+osbg3KynNF261ZL2C+bDg8rJRFHVg5Cch9W1tKhj/4KlNNdvnKgfQ8z8\nLi8pyJjc2egsJnf7fFNMNlcE80NeLe9BSf5tUwjRgB1f7fYYOncyRVrNNSR43DcHAxaKV1xTvzR9\noXUdemBTO96Zn0CiKGNyCnhXJovIGuAeYAXmLber6i9FZDlwH7AO2AtsUdXj7QxGCs5MD4njAu1K\nj2tc98mBT4aWLxy3E2MXPDgAWFguTkT8HI31cuBJJU917i+aIrnhAisafxZPs3qkGE+0toNWmBwB\n31fVK4Drge+IyBXAHcBOVb0E2OnrGRbBuzJZVQ8CB/3/aRF5FXvzaTNwox/2e+Ax4Edtjcb9nzTU\nQOZWWD7hD9Mr40HZMhYMHvHF7AsiaqnOOLlUQ7Vat/rfN+zNqfHVz/uWEVskyOQzevCJyDrgGuAZ\nYIX/AAATmDtZ7Jy6V8zej2jZyCLSD9wP3K6qU3LKU1dVVaSRf7V9rb9iFjO5Wn/YplX2Wz5y7Iq6\n4yoFZ6kvqp6t04CTL0mGtey9LRoKCsOindwXmNqIIzxiXey+vB20pC5EJI8Z+I+q+mfffEhExn3/\nOHC47dGcp2hFXQjwO+BVVf35KbseAL4O/NSXf0lqUJWuem060m0++Ym3LHO2IbJXEMKS++KYta6F\nRd/ZhkSew2joKzfrU1s5y5MEnqyPSwak0H4BYivu4uPA14CXRCR+Z/bHmHF3iMg3gLeALW2P5jxF\nK+riSU7GYI1o40XpRfryUippeCfmlhF78j+y60oASpdblFZc5TMiXpLVPek6OwdVd8VxPlkbljFC\nrxi4LG+6uTaF5TmPJJBFfCmgo3IXtaK/Un0udyy0yc+RXTE9jeq5eVcMTpXSmCmH/IySC7wUYGgQ\ngGjI5GOsXOJZmBXPmaqI5/qijfZKWvj8a3a8l9S2g4zJKaCjmFybjXAJEGxYB8BQ8DgAI3c9XXf4\nYAtN1rLAB6xoJVYX8TK30/z9GvfJ+24yxq99ypx10FIvp0fG5BTQUUye3mgadX7QxExchvX5e34A\nwDqeXvzEBPCtHVsBGNprHM9dbHN/mkD9RcbkFNBRTO6a9sogr5vIHbdIb/WjTTJhCX4IZMN9FvHJ\nvBcm5trPvsXImJwCUn2BXUSOADPAZGqdJo9R3jn+i1R1rNkJqRoZQESeU9VrU+00QSxl/Jm7SAGZ\nkVPA2TDy9rPQZ5I44/Gn7pPfj8jcRQrIjJwCUjPyufhBaxFZIyKPisgrIvKyiHzPt/9ERA6IyAv+\nd/Np20nDJ5+rH7T2WfhxVd0lIgPA88At2HxmUVV/1ko7aTG59kFrVV0A4g9adzRU9aCq7vL/p4G4\neuqMkJaRF/ug9RkP9myioXoKYJuIvCgid4vIcNMTyR58LaGxegr4DbABuBqrE7zzdOenZeRz9oPW\ni1VPqeohVa2oahW4C3OHTZGWkWsftBaRLuyD1g+k1PeS0ax6Ki5Pc3wR+Nfp2kklaX8Of9C6WfXU\nrSJyNTYfuxf45ukaycLqFJA9+FJAZuQUkBk5BWRGTgGZkVNAZuQUkBk5BfwfYON04pVnA+IAAAAA\nSUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 72x72 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Predicted class: 4, confidence: 98.32%\n",
            "Actual class: 4 \n",
            "\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAFkAAABYCAYAAACeV1sKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAMx0lEQVR4nO2ce3Bd1XXGf+teXb0l63Et2dbDkpBs\nMMXYuJjEdkjGmQQwDZi2ZAgp5Y82aWZCQjJ0hkybzHSa6UyHNnT4J6R08oI2QKaliScwvFxehoSA\n8QsM2MY2tmTrZUnWW7qPnT++fXmoyJbuVQ6X+HwzmnPPY++zz9J31v722msfc84R4veLyIfdgHMB\noZEDQGjkABAaOQCERg4AoZEDQE5GNrMrzexNMztkZt9aqEb9ocGy1clmFgUOAJ8BOoGXgC845/Yv\nXPP+MFCQQ9n1wCHn3GEAM3sAuBaY1ciFVuSKKcvhlh7mtzP4kahT3emyNOavcUn9iExpG+sZy/3+\nMzDJGNNuymY7n4uRG4Dj79nvBC6beZGZfRn4MkAxpVxmn87+jpGo6ozoeVwy+b7T3TduAGBs/QSR\naAqAxFAxABUH9ahLv/fCzAZqO9sbfbbzwItu+xmbnYuR5wTn3D3APQCVVpPbGD4tw7n0jMOb1gCw\n5/bvA3BL12WkPd1Hk4UA/OBPngTgT+/SP+Kdf1DGeLMZcwHCDrl0fF1A03v2G/2xEDOQS8dXgDq+\nTyPjvgTc6Jx7bbYylVbjcnIXmXtfehEA4w0lAJScmAAgUVUEQPnfd9JcOgjA4dFaALp+2aI2HNPb\ncHKDmNvx7d0ApCcns27Pi247w25g4X2ycy5pZrcAjwFR4EdnMvC5jJx8snPuEeCRBWrLrEhuXgfA\nka1qbqxezC36rfYnq8oBKDkllu57q5HO+CIAxibE7uZdKpMqUucZG5GvPvjdtQA0PKuy5bvk8ZKd\nC+f5whFfAMjaJ2eD+frkAz9YD8DSp8SFwfO1LT6l8/F9YudQm2Ta6c+OA1C4pwzn6ZMSkUmWS5I0\nPS5VMVWlt6BXt8CJ4BSMybW2/vewju86uwc8m08OmRwAfu86ORu4jdK9t1/+MAD/+5XFAFSuuxAA\nS8h/pve+AcDiN+oAmKpuB2DpnS8QWX0+AKkyUblvnUaDhY+9DEBJZSUA5cdbAYgd7gZgeEMLAIf+\nVj77vC/m/jwhkwNAXjK5Z30pANv+fCMAJ2+LA7D4qk4AugakHNzBj6tAu+IR962/C4CbK25lsnka\ngKr4KABXNe8C4JlejfgGO8SviVZdVzDQBsCiN1Vl7aPFC/Y8IZMDQF4yOX35EACjR6oAqHtFo7Gh\n0w0A1IxKEVXt7AFgvEOjuht6bwGg/clxBldoNJiOVQPwwIVicFuvmDtWL6Ym+2LAu+oig9FGiYV4\no+6Zi24OmRwA8pLJ8XL52MEmMXlJl/Rw+QnRLVniudEnwVwYrwDAEmJvdPdByqpWAVAwLiUy3C6V\nMb1Ij1w8KN1cMCnG1j+qqO3Rv2gGYLw1AUBqid4EQibnN/KSyb3DikU4EZmjn9N+cb9YlxRhKeta\nDsDpdqmRouYRlVvVxqlV8rVFA3rEaItURsXdGslNtciP914i39xzpaK2kfXqD6IT0smpYpXPhY0h\nkwNAXjLZ7ZYObrlbonXwMx0ADLeKyVPVUhdjjSV+X8cXlcl3T9XWkCj3dUV1rqHmNACTbRo9DqyU\nj868FZMiNtFden3KdDmxXimYVA7PEzI5AOQVkzMxi83X7ATgyI810stE1NK+tYXDfuZ5RPwqPK0L\nSmNSBIkiIx3z0cUJXXtiUG/H8gFp7oif+7O0zkentV3+sCg83qhYx4kr6wGoP/BW1s8VMjkA5BWT\nExVSBJsqDwDwxFcuAeDSza8DcOyOFQBU7O0FIHlU2ra4WP712DK9CU2P7mRZcrXOdSvG3DegqJvb\n9WsA6jvlm6mrAWDD/XsA+OmKjwFwzUpF636x/2Jdn8NzhUwOAHnF5JKXDwNwx7/dAED7L7S/r1+j\nt5GtU7pws3hVemIZAObzMDZdp0jbviNrOd3mo2xXSGZ8cv2rABwYVP7NSJNGjxkVcpl3/NWPS248\n9Zyua/OzL7kgr4w8tboFgOF2dVqRzyn8OLlY+xaVNTPBnJSPRprXV1dV7wPgmVVrSHdoaF5apOmm\niM/pSsfUwY01qq50tTrLh47ILdR06/rBFXJdfWs10Knfkf1zhe4iAOQVkwdXSlZFvVdIedb92RbR\n6Nl/UpB+0U5NFaV7+wGwqKj9nYm/BKD13/cx8QlNPxVM6Nye85QQU/szdXzV2zVl5erV8U3fqQ6y\ne60CQslS/zYlvMSr1/Wpnt55P1fI5ACQX0xeI39YMKhmTYlkfKlGmZg7kpJXJ7b44P3rGqwkKr2T\n3qjgTmJnB4Mr5VOnF8V8XXLc9U2NAPRtVkAoOi3G9nTJ9xasli9PnVQH6Gr1WqWXeckXMjk/kVdM\nrvVpV3XPyOcOrZNU+9qR6wEo6tfUUf+Fui5VJI44n69s5hVE1CgY1+/JuK88Q6eYyqZFcKKqEjem\n44UVYm7xa6ozUSEJE+nUwCebQFHI5ACQV0weWC32Vb+pYE7/xWJTf7f8YbxRrJpokO8e7VHzExW6\n7tIlYtve5osYaVGdyWViZqxYZaabpB5Gm1SmyGdXVRwS30YK5Jvrjks/n27xlPfDb/r65v1cIZMD\nwFmZbGZNwL0oRuKAe5xzd5lZDfAg0AIcBT7vnBvMpTE3feo5AO6zTQBsv/ZfALhx/80AVO/QxKml\npQyqnn8bgOl2+e7/O/8CAFbuGcZFFBCaGBb7M6PEyLPSyfG4Mg2LT4mx4/XS6A9+XQky32xXP3BN\n7REAtvFJlcsiA3suTE4CtznnVgEfA75qZquAbwHbnXMdwHa/H+IDcFYmO+dOAif97xEzex2tfLoW\n+JS/7KfA08DtuTRm+z+KwSsf1yq1zWXfAKDkiFhWFZcOTnk36cakaWP9PmW2V34z8lYn8Yim9keb\n5WOnKiOZBwKg/LAmVtOlqixVKN+85bFbAVj8a5nmV76eJU/I379/vdXcMK+Oz8xagLXAi0C9/wcA\ndDNLyHXmErNzEXM2spmVA/8DfMM5N2z2bs6zc85ZRqTOwHyWmJ1uk+Ms8HGHTNSt+oC2x6/UJOeK\nLQcB2LtGIVC3VFNKOy6XD99YdRulDWLq1a0vAhCPKV3gJ0uvACBxkd6CxdVKERh7WgQojeutGK+X\nwsno6ZRfHsHxzjM9wgdiTurCzGLIwP/lnHvIH+4xs6X+/FJg/uPNcwRzURcG/BB43Tl353tObQNu\nBv7Zb3+Za2OS3puM16lZ1bVSE1XPanKz6nkxvfeg4swdDytIP7VJyeEbJr6p4/dP0r9a6mJbvV8c\nGdVL1PbDQwBMXixfO1Gr6FrD/YqPHPyJFgEVljnfJm1TZaJ0Npp3Lu5iI3ATsM/Mdvtjf4eM+3Mz\n+yvgbeDzWdz/nMBc1MUO3l0yPhO5r3x8DxLlYs1og263tVHLFXb5tK1T12v2IqN5KwulOsaWaVt+\nILN0d5qYXCulryrakPH3mXhwcZdGfmNLpEgK2lpUYNRHABulnwvLFNxIleoe2TA5HPEFgLyKXdT4\n0dTip9WDP9BxKQAdk/JSmRhFJOHn/OJiYd96qY/CAXHm+BVl7/jS8rfF4Kl1UhvRVUorGGuTWhhZ\nrjqr9+ttWbJD+wMXiLmZRZUlrx0FstPJIZMDQF4xufKoImYnrtbsxfIGn3jtP8Ew9EfiUXG3jwlX\naPaiqlkjwbEhHynDvZOmlSwRM0uK5WOTVSqTKPMpAz6il/kUQ3RK5abqdLwsk3h+sifr5wqZHADy\niskFzysBZfKrKwH4dtuvALgDzTR33Cumn9yoZltSvri2TFIiOaD819HWFOZnmVM+Nba6RKNCS78/\nVlEwJKa6l5SzMX6JZsQvOF8RvrePtaiCmV8ymQdCJgeAvGKyS0iTVm5TT/+lnr8GoAPFH+wFJQUu\n858Rysy3Df2n2Nf4gjTwsa11lHeJeZVvKUYx0L0EgJLfKJ5cu18jwqp7h9/XhrrfyL/3prVUou5Y\nZhIw+4X+IZMDQKCfYjCzPmAM6A/spguPOP+//cudc4tnKxCokQHM7GXn3B8HetMFRDbtD91FAAiN\nHAA+DCPf8yHccyEx7/YH7pPPRYTuIgCERg4AgRn5o/hBazNrMrOnzGy/mb1mZrf64/9gZl1mttv/\nbTljPUH45I/qB639LPxS59wrZlYB7AS2ovnMUefcv86lnqCY/M4HrZ1z00Dmg9Z5DefcSefcK/73\nCJDJnpoXgjLyB33Qet6N/TAxI3sK4BYz22tmPzKz6jOVDTu+OWBm9hRwN3AesAblCX7vTOWDMvJH\n9oPWH5Q95Zzrcc6lnHNp4D+QO5wVQRn5JaDDzFrNrBC4AWUg5TVmy57KpKd5XAe8eqZ6Agnaf4Q/\naD1b9tQXzGwNSoo/CvzNmSoJh9UBIOz4AkBo5AAQGjkAhEYOAKGRA0Bo5AAQGjkA/A5EKyEOr6sD\ntwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 72x72 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Predicted class: 6, confidence: 93.678%\n",
            "Actual class: 6 \n",
            "\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAFkAAABYCAYAAACeV1sKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAFy0lEQVR4nO2cTWxUVRTHf2c+2lI+mmIttHxLiErE\nIFRQcWPEhGAMGgKBhWFhBD9INHEhMZq4dKGuNBqMJCxIjFET0SDENLIwUSxiAwKiTVO02A8KwmBL\nW2fmuLivpFNm2mln5rQz3F/S9L137zv39D/nnfte57wrqoqnsIQm24FbAS+yAV5kA7zIBniRDfAi\nG5CTyCKyQUTOiUiLiOzJl1Olhkz0PllEwsDvwGNAO9AEbFfVM/lzrzSI5HDuGqBFVVsBROQTYBOQ\nUeQyKdcKpucw5NSkn14GdUAyteci8jzgr2H77cDakZ1EZCewE6CCStbKozkMOTU5po2jthd84lPV\nvaraoKoNUcoLPdyUJBeRLwALhu3PD455RpCLyE3AMhFZIiJlwDbgYH7cKi0mnJNVNS4iu4EjQBjY\np6qn8+ZZCZHLxIeqHgIO5cmXksU/8RngRTbAi2yAF9kAL7IBXmQDvMgGeJEN8CIb4EU2wItsgBfZ\nAC+yAV5kA7zIBuT0/2QrIvPqAbh+dx0A8RlhAGILnfs1J/sBaHvWlTfMPxCh8nwMALl0xZ3T2ZVq\nNORskEykH1SCL5/zUFrsI9kA+0iWYeUJI6JEIs4djccB6N3sKgwiu1wUPl5/FID6qIvOK4lKAL7q\nvBeArdWuQuHwonXEnnM2P1zxNQCvPv8CAGWHm1zDiAgOVVS4w/39aX3LBR/JBky4TGsizJLZuja0\nHgm7fKiJEfkw8CXxyCrn3OsXAWhpmQtAdbOL9NofXb4N91wFoH3zQgB61/QB0LDoT7r6ZgLwRN0p\nAFZNawPgs8v3A/DTe26M6v0/5Px3HdNGYno5YwWRvchZVBAd+bsZgMdXbwAg3tE57rFC99wFQOsb\nZQBsufMXAKoi7oO4MFDtxjq4BoDKTqdD/+xUrTS41jWYJwmak1HXf0HjAMeb3icWa88osk8XBphO\nfFJeTnjxUmIragCI9iadE71uopOki44l3zQAsHxaDwADG90l3l/twile4YImGXV2Nez2r6weBKCm\nNsbVa64kTFrc5Hjo6MMAVJ9zfSpaXSoKb3M2/nlowG1cc0Yl7mxqyPmkkeCKD8IyMsvZGTxeRnIo\nyjPgI9kA05xcFa3VB2u2cHn9HQBcr3WfcW+980FcYDOt20VR39xU35LlrkNo0LVH+iSlf1XrfwBM\n/+0i8da2cfkWrrnNbQw6G5RFUztIEI+h1NSb6Ooec+LzkWyAaU7WeJxEVzdVB7oBqCrQOPEJnJPo\nuZR3P4bwkWyAF9kAL7IBY4osIgtE5DsROSMip0XkpeD4bBH5VkT+CH5XF97d4iSbSI4Dr6jqcuAB\n4EURWQ7sARpVdRnQGOx70jCmyKraoaongu1rwFncm0+bgP1Bt/3Ak4VystgZ1y2ciCwG7gOOAXNU\ntSNo6gTmZDgn5RWzW5GsJz4RmQF8DrysqrHhbeoeG9M+OvpXzLIUWUSiOIEPqOoXweEuEakL2uuA\n7sK4WPxkc3chwMfAWVV9d1jTQWBHsL0D+DL/7pUG2eTkdcDTwCkRaQ6OvQa8BXwqIs8A54GthXGx\n+BlTZFX9nhvfB9xE6b0oXQD8E58BXmQDvMgGeJEN8CIb4EU2wItsgBfZAC+yAaZ1FyJyEegFeswG\nzT813Oz/IlW9PdMJpiIDiMhxVW0wHTSPTMR/ny4M8CIbMBki752EMfPJuP03z8m3Ij5dGOBFNsBM\n5GJc0HqU6qk3ReSCiDQHPxtHtWORk4t1QevgW/g6VT0hIjOBn3FFPFuBf1X17WzsWEXyjQWtVXUQ\nGFrQekozSvXUuLASOd2C1uN2djIZUT0FsFtETorIvrGKLf3ElwVpqqc+AJYCK4EO4J3RzrcSuWgX\ntE5XPaWqXaqaUNUk8BEuHWbESuSiXNA6U/XUUHlawFPAr6PZMXkxp4gXtM5UPbVdRFbiiizbgF2j\nGfGP1Qb4ic8AL7IBXmQDvMgGeJEN8CIb4EU24H9/0OMUthZ8RQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 72x72 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Predicted class: 5, confidence: 99.913%\n",
            "Actual class: 5 \n",
            "\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAFkAAABYCAYAAACeV1sKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAIQElEQVR4nO2cbYxUVxnHf8/Mzsy+sm/Asl1woRYD\nqBVaCkWLaW0whsZSGyQSY/hgUk3atI1+EP3kF6sfrNFGY6SxEWOjbawJ/UDSBGIrVaFQpLwUaBGW\nArvAwsK+scPszH388NypzDA7u+zsnn3h/BJy576d8+yf/33OOfeee0VV8YwvkYkO4HbAi+wAL7ID\nvMgO8CI7wIvsgJJEFpGviMhxETkhIpvHKqjphoy2nywiUeADYA1wFtgLbFTV98cuvOlBWQnnrgBO\nqOpJABH5C7AOGFLkuCS0nKoSqpycJOknpddlqP2liNwCnLlh/SywMv8gEXkCeAKgnEpWysMlVDk5\n2aM7i+4f94ZPVbeo6nJVXR4jMd7VTUpKEfkcMO+G9bnhNk8epYi8F1goIgtEJA58A3h9bMKaXow6\nJ6tqWkSeAt4AosBLqnpkzCKbRpTS8KGq24HtYxTLtMWP+BzgRXaAF9kBXmQHeJEd4EV2gBfZAV5k\nB3iRHVDSiG/CkfAWroReCTLDnjKwbgUAFdveKanqaF0tAJmr3cMe653sgKnt5OyjM73ZwR/+4V4A\nWlsuAdC++w4AFq5uAyA4dhcAmeMnChYtsbgVPZjK2d72k1UAPLb23wAc+lI90h0tGqZ3sgOmhpMj\noVM0CJeFH/5G7l4EwE+3beXxf9wDwJmDzQDEMpa/P7paB0Djr68BEF9TuMp8B194+vMAfP2RXQAs\nSHQCsO+eewn2FH/i453sgMnh5GwvIevQ/PW8XkOkvNwOW2BPv57b/icA1r92PwAb/vwsTYfs3J75\n5qN0pa0PHDcnt6zuAOCHbZZbH3/5ewAs+GuP1ZFKA9DxUCMAD26y3shAJgbAi6ceAKDx1OWPjx0K\n72QHjHpyy2iYIQ06mikBya+uyFlv2fwhAIc75wCQ2Gbu1NAy/S3CYG2Qc06mIlwPL5L4JcvzzSvN\n0d9tfQuAy5lqAD66bg5eVnkagDeufAaAN/cvBmDV5yyGrmda2H3kd/T0tw8578I72QGTIyfnEVm6\nBIBUYwUAZzcMArDyzjYAzjz/KQAqqswj1+aYidKVdn5QpkTCCT1Zd0ci9iMoN0cH1g2me5v1n59L\nbASgd6Hl1znzLwPwttwJwNW37Kqps5TNow8fAGBrsgkZJhm4FbmyHFn0aU58swaA8kv2h2fCHtBg\ntUWrMVvGr9j+ul22fvg/dqkm77N1CTNArNeWErY/EoVUvTWWlbP7rYyqAQBqE0kAZsRtOTNu+88n\nLab9Jz8BwNXdTRbDVSszqLdl4pGLABy8Zo1u17J60u1+MDLhOHXy9VkRjj9ZwcrFHwDQkyrP2T8Y\nmCMu9Frj0z/T9g/OMKsnLlkKKOuz5UCrpZHYYhtY3N1kjdhdVZ00xey6nlVmy8rI9Zy6usIG7sKg\n3ejZfsQatkc/+x4Av1yzz2JWq+OFKzbQCcL8kwysK1d/rI/oQG4jm493sgOcduFq6ubqsgee5uxD\ndgFJi+XJxro+AGZX2bKyzIa07X3mshlhHu3otbz54B12U6clYQnzWtiKZd11sn8mNTE7Z8dxc2Dd\nLrsqko12FSxea1dT7+pLBWONNjZYjDXVOdu12xoAHbDYI/V1/KvzFbpTF30XbiJx6uTa2Gxd1bAe\n6syRmf+ezg0mZg6PVNtE8cElrQBca7ac3P5l6z4077Dj6t85D0DQHi6TyRHHEl1oXbMry2cD0LDH\nyiAI82vaeifa1xcGZ36U2jD2+nBZHWfv/t/Q03vOO3kicdq70HSaTGcn0QbLtX3r7wMgCKOIDNpV\nVd5ljo3vPgZA7Is2OFn0grkqOGjbg+yNorl2O1Pq/v+qRKbK8vNgZVh46LNI2uqInuoCoOy6rZ9f\nY2XMetdybhC3no6kLTdrNLzRVGPlpmpsf8XF3FuihfBOdsCwOVlE5gF/BJoABbao6q9EpAF4BZgP\ntAEbVPVKsbKGu0GUHU5nquI523tbzbHll83hyUZzZzph9kz0WB6NprIjQSXWZ8dG+62fG8TCUVnE\nzskkbD2atONS9VZnotPyuobHp2pjObHEuwfDcs3BwXtH2RPsoEe7SsrJaeD7qroEuB94UkSWAJuB\nnaq6ENgZrnsKMGxOVtUOoCP83SsiR7E3n9YBD4aHbQXeBH5QSjDBAXs7Ld8SM/6Zux5n5GSv0/wy\n8//w/AdIMsT2LMXHeMXrKoqIzAeWAXuApvA/AOA8lk4KnZPzitntyIgbPhGpBl4DnlXVnhv3qSX2\ngsndv2I2QpFFJIYJ/LKq/i3cfEFEmsP9zcDF8Qlx6jOsyCIiwO+Bo6r6ixt2vQ5sCn9vAraNfXjT\ng5Hk5C8A3wIOiciBcNuPgJ8Br4rIt4HTwIbxCXHqM5Lexdvc3DhnmX4vSo8DfsTnAC+yA7zIDvAi\nO8CL7AAvsgO8yA7wIjvAi+wAp0+rRaQT6AcKT3aYGszk5vhbVXXWUCc4FRlARPap6nKnlY4ho4nf\npwsHeJEdMBEib5mAOseSW47feU6+HfHpwgFeZAc4E3kqftBaROaJyN9F5H0ROSIiz4Tbfywi50Tk\nQPhvbdFyXOTkqfpB6/ApfLOq7heRGuBd4DHseWafqv58JOW4cvLHH7RW1RSQ/aD1pEZVO1R1f/i7\nF8jOnrolXIlc6IPWtxzsRJI3ewrgKRE5KCIviUh9sXN9wzcCCsye+i3wSWApNk/w+WLnuxJ5yn7Q\nutDsKVW9oKoZVQ2AF7F0OCSuRJ6SH7QeavZUdnpayNeAw8XKcfI6wxT+oPVQs6c2ishSbJJlG/Cd\nYoX4YbUDfMPnAC+yA7zIDvAiO8CL7AAvsgO8yA74HxfwsPwb9Ot6AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 72x72 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Predicted class: 7, confidence: 99.998%\n",
            "Actual class: 7 \n",
            "\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "TKhCzP65hGyS"
      },
      "source": [
        "## Conclusion and next steps\n",
        "\n",
        "In this tutorial, you learned how to customize `adanet` to encode your\n",
        "understanding of a particular dataset, and explore novel search spaces with\n",
        "AdaNet.\n",
        "\n",
        "One use-case that has worked for us at Google, has been to take a production\n",
        "model's TensorFlow code, convert it to into an `adanet.subnetwork.Builder`, and\n",
        "adaptively grow it into an ensemble. In many cases, this has given significant\n",
        "performance improvements.\n",
        "\n",
        "As an exercise, you can swap out the FASHION-MNIST with the MNIST handwritten\n",
        "digits dataset in this notebook using `tf.keras.datasets.mnist.load_data()`, and\n",
        "see how `SimpleCNN` performs."
      ]
    }
  ]
}